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Executive  Summary 


As  shown  in  Figure  1,  the  act  of  acquisition  planning  is  one  of  defining  and  maintaining  an 
overall  approach  for  a  program.  The  acquisition  planning  process  guides  all  elements  of  pro¬ 
gram  execution  to  transform  the  mission  need  into  a  fielded  system  that  is  fully  supported 
and  delivers  the  desired  capability.  The  goal  of  acquisition  planning  is  to  provide  a  roadmap 
that  will  be  followed  to  maximize  the  chances  of  successfully  fielding  a  system  that  meets 
users’  needs  within  cost  and  on  schedule.  Acquisition  planning  is  an  iterative  process;  feed¬ 
back  loops  impact  future  acquisition  planning  activities. 


Figure  1:  The  Context  of  Acquisition  Planning 

Many  programs  struggle  during  acquisition  planning  because  even  though  a  wealth  of  infor¬ 
mation  exists,  the  process  itself  is  not  clearly  defined  or  understood.  Acquisition  planning  is  a 
powerful  process  that  can  help  define  the  most  promising  acquisition  options  that  best  miti¬ 
gate  risks.  In  turn,  developing  an  acquisition  strategy  is  a  key  component  of  acquisition  plan¬ 
ning  that  provides  a  means  of  addressing  program  risks  through  the  program  structure.  In 
most  cases,  however,  acquisition  planning  and  acquisition  strategy  development  are  done  in 
an  ad  hoc  manner,  without  consideration  of  the  internal  and  external  factors  driving  the  pro¬ 
ject.  When  developing  an  acquisition  strategy,  it  is  important  to  understand  these  factors  be¬ 
cause  they  often  correlate  to  what  drives  the  program’s  risk. 

For  software-intensive  systems,  the  acquisition  strategy  must  acknowledge  the  type  and  ex¬ 
tent  of  software  risk  to  the  program  and  include  plans  to  actively  mitigate  those  risks.  Pro¬ 
grams  need  structured  ways  to  reason  about  software  risks,  formulate  acquisition  strategies  to 
mitigate  software  risk,  and  evaluate  their  current  acquisition  strategy  in  an  ongoing,  system¬ 
atic  manner. 

The  acquisition  strategy  guides  decisions  made  throughout  the  life  cycle  of  the  program.  It 
should  be  based  on  reasoning  about  both  internal  and  external  complex  factors  with  regard  to 
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the  system.  When  developing  an  acquisition  strategy,  it  is  important  to  understand  the  pro¬ 
gram’s  driving  factors.  These  drivers  are  conditions  that  impact  program  risk.  For  example, 
the  skills  of  the  personnel  working  in  the  program  office,  the  number  of  stakeholders,  and  the 
number  of  system  configurations  that  need  to  be  maintained  are  all  drivers.  Drivers  can  influ¬ 
ence  the  acquisition  strategy  independently  or  there  can  be  interactions  between  drivers  that 
affect  program  risk.  For  example,  unstable  requirements  usually  influence  overall  costs. 


Figure  2  shows  the  relationships  among  the  program’s  drivers,  its  risks,  and  its  acquisition 
strategy. 


Figure  2:  Relationships  Among  Drivers,  Risks,  and  Acquisition  Strategy 

The  research  results  presented  in  this  report  support  a  more  systematic  approach  to  reasoning 
about  software  risk  on  a  program.  The  methods  and  techniques  presented  contribute  to  the 
work  that  focuses  on  developing  an  acquisition  strategy  from  a  sound,  systems  engineering 
approach. 

This  report  outlines  research  performed  to  help  acquisition  planners  more  systematically 

•  profile  conditions  that  impact  program  risk  (drivers) 

•  identify  sources  of  potential  software  risk  early  in  the  program  and  throughout  the  pro¬ 
gram  life  cycle 

•  develop  an  acquisition  strategy  by  decomposing  it  into  the  strategy  elements  that  com¬ 
pose  it,  addressing  them  individually  and  then  collectively 

•  reason  about  their  acquisition  strategy’s  ability  to  mitigate  software  risk  in  an  ongoing 
manner 
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The  report  introduces  a  taxonomy  of  strategy  drivers  and  strategy  elements.  It  also  provides  a 
method  for  performing  a  comparative  analysis  of  the  strategy  drivers  and  the  resulting  strate¬ 
gic  choices  for  the  elements.  For  a  program  with  an  acquisition  strategy  already  in  execution, 
the  method  can  be  used  to  perform  a  strategy  analysis.  The  method  uses  a  technique,  slider 
bars,  to  support  a  more  systematic  approach  to  reasoning  about  software  risk  on  a  program. 

For  example,  to  use  this  method  and  the  slider  bar  technique  to  develop  a  strategy,  acquisition 
planners  would  perform  the  following  steps: 

1.  Define  the  objectives  of  the  acquisition. 

2.  Identify  and  evaluate  the  factors  that  drive  the  program. 

3.  Decompose  the  strategy  into  individual  strategy  elements. 

4.  For  the  selected  strategy  elements,  identify  the  potential  strategic  choices  and  rank  them 
in  order  of  their  risk  mitigation  capabilities  for  your  program. 

5.  Evaluate  the  strategy  drivers  for  the  program  to  identify  those  that  influence  the  strategy 
element  through  the  introduction  of  risk  that  may  be  mitigated  by  the  strategy  element. 

6.  Define  the  relationship  between  the  risk  generated  by  the  strategy  driver  and  the  risk 
mitigation  capabilities  of  the  strategy  element. 

7.  Map  the  driver  evaluations  from  Step  2  to  the  strategy  element  using  the  slider  bars. 

8.  Choose  the  strategy  that  best  mitigates  the  risk  elements. 

9.  Identify  residual  risks. 

The  Acquisition  Strategy  Development  Tool  (ASDT)  is  a  customized  Excel  workbook  we 
created  to  help  acquisition  planners  work  through  the  method  and  techniques.  The  ASDT  is 
provided  so  that  acquisition  organizations  do  not  have  to  develop  slider  bar  templates  from 
scratch.  The  ASDT  is  available  for  download  on  the  SEI  Publications  Web  site  at 
http://www.sei.cmu.edu/publications/publications.html.  Note  that  acquisition  planners  can 
also  apply  the  method  and  techniques  discussed  in  this  report  without  using  the  ASDT. 

Our  research  focused  on  identifying  and  mitigating  software  risk  in  a  program  during  the  ac¬ 
quisition  planning  process.  However,  the  methods  and  techniques  we  present  can  be  applied 
to  acquisition  planning  areas  other  than  software  risk. 
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Abstract 


The  goal  of  acquisition  planning  is  to  create  a  roadmap  that  a  program  can  follow  to  maxi¬ 
mize  its  chances  of  successfully  fielding  a  system  that  meets  users’  needs  within  cost  and  on 
schedule.  Developing  an  acquisition  strategy  is  a  key  component  of  acquisition  planning  that 
provides  a  means  of  addressing  risks  through  the  program  structure.  Programs  need  struc¬ 
tured  ways  to  reason  about  software  risks,  formulate  acquisition  strategies  to  mitigate  soft¬ 
ware  risk,  and  evaluate  their  current  acquisition  strategy  in  an  ongoing,  systematic  manner. 

This  report  introduces  a  taxonomy  of  strategy  drivers  and  strategy  elements  and  provides  a 
method  for  performing  a  comparative  analysis  of  the  strategy  drivers  and  the  resulting  strate¬ 
gic  choices  for  the  elements.  The  primary  audience  for  this  technical  report  and  the  accompa¬ 
nying  Excel-based  tool  is  program  managers  of  government  acquisition  programs.  The  main 
prerequisite  for  successfully  using  this  information  is  a  working  knowledge  of  government 
acquisition  practices. 
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1  Introduction 


1.1  Purpose 

The  goal  of  acquisition  planning  is  to  provide  a  roadmap  that  will  be  followed  to  maximize 
the  chances  of  successfully  fielding  a  system  that  meets  users’  needs  within  cost  and  sched¬ 
ule.  An  acquisition  plan  guides  all  elements  of  program  execution  to  transform  the  mission 
need  into  a  fully  supported,  fielded  system  that  delivers  the  desired  capability. 

Many  programs  struggle  during  acquisition  planning  because  even  though  a  wealth  of  infor¬ 
mation  exists,  the  process  itself  is  not  clearly  defined  or  understood.  Software  acquisition 
planning  is  a  powerful  process  that  can  help  define  the  most  promising  acquisition  options 
that  best  mitigate  risks  in  the  development  of  software.  Developing  an  acquisition  strategy  is 
a  key  component  of  acquisition  planning  that  provides  a  means  of  addressing  program  risks 
through  the  program  structure.  In  most  cases,  however,  acquisition  planning  and  acquisition 
strategy  development  is  done  in  an  ad  hoc  manner,  without  consideration  of  the  internal  and 
external  factors  driving  the  project.  When  developing  an  acquisition  strategy,  it  is  important 
to  understand  these  factors  because  they  often  correlate  to  what  drives  the  program’s  risk. 

For  software-intensive  systems,  the  acquisition  strategy  must  acknowledge  the  type  and  ex¬ 
tent  of  software  risk  to  the  program  and  include  plans  to  actively  mitigate  those  risks.  Pro¬ 
grams  need  structured  ways  to  reason  about  software  risks,  formulate  acquisition  strategies  to 
mitigate  software  risk,  and  evaluate  their  current  acquisition  strategy  in  an  ongoing,  system¬ 
atic  manner. 

During  the  past  year,  the  Acquisition  Support  Program  at  the  Carnegie  Mellon®  Software  En¬ 
gineering  Institute  researched  more  systematic  approaches  to  reason  about  software  risk  on  a 
program.  Specifically,  we  explored  answers  to  the  questions  “Are  the  methods  and  techniques 
arising  from  a  sound,  systems  engineering  approach  ones  that  can  be  applied  to  acquisition 
planning  and  acquisition  strategy  development?  Will  they  enable  programs  to  better  reason 
about  software  risks,  formulate  acquisition  strategies  to  mitigate  those  risks,  and  evaluate 
their  current  strategy  in  an  ongoing,  systematic  manner?” 

This  report  outlines  the  research  that  was  performed  and  is  designed  to  help  acquisition  plan¬ 
ners  more  systematically 
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•  profile  conditions  that  impact  program  risk  (drivers) 

•  identify  sources  of  potential  software  risk  early  in  the  program  and  throughout  the  pro¬ 
gram  life  cycle 

•  develop  an  acquisition  strategy  by  decomposing  it  into  the  strategy  elements  that  com¬ 
pose  it,  addressing  them  individually  and  then  collectively 

•  reason  about  their  acquisition  strategy’s  ability  to  mitigate  software  risk  in  an  ongoing 
manner 

The  report  introduces  a  taxonomy  of  strategy  drivers  and  strategy  elements.  It  also  provides  a 
method  for  performing  a  comparative  analysis  of  the  strategy  drivers  and  the  resulting  strate¬ 
gic  choices  for  the  elements.  The  method  proposed  is  a  bi-directional,  graphical  method  of 
examining  and  analyzing  the  relationships  between  the  drivers  and  the  strategic  choices.  The 
method  is  bi-directional  in  that  it  enables  a  program  to 

•  analyze  strategy  drivers  and  choose  strategies  for  each  strategy  element  to  minimize  the 
risks  induced  by  those  drivers,  or 

•  choose  a  strategy  for  each  strategy  element  and  analyze  the  risks  induced  by  the  pro¬ 
gram’s  strategy  drivers 

The  method  uses  a  technique,  slider  bars,  to  support  a  more  systematic  approach  to  reasoning 
about  software  risk  on  a  program.  This  work  extends  a  concept  introduced  by  the  Department 
of  the  Navy  representing  strategy  elements  as  a  continuum  or  sliding  scale,  much  in  the  way 
the  volume  control  on  a  stereo  can  range  from  soft  to  loud  [Department  of  the  Navy  01]. 
Slider  bars  can  be  used  to  graphically  visualize  a  program’s  strategy  drivers,  acquisition  strat¬ 
egy  element  strategic  choices,  and  the  relationships  among  the  drivers  and  choices.  The  Ac¬ 
quisition  Strategy  Development  Tool  (ASDT)  explained  in  Section  1.3  is  provided  to  assist 
program  offices  work  through  the  methods  and  techniques. 

Our  research  focused  on  identifying  and  mitigating  software  risk  in  a  program  during  the  ac¬ 
quisition  planning  process.  However,  the  methods  and  techniques  we  present  can  be  applied 
to  acquisition  planning  areas  other  than  software  risk. 

1.2  Document  Structure 

Section  2  discusses  the  need  for  acquisition  planning  and  the  roles  of  the  acquisition  strategy 
and  acquisition  plan  in  the  acquisition  planning  process.  It  defines  key  terms  and  highlights 
the  relationship  between  acquisition  planning  and  software  acquisition  planning  including  the 
challenges  posed  by  software. 

Section  3  introduces  a  taxonomy  of  strategy  drivers  to  help  programs  identify,  evaluate,  and 
understand  acquisition  strategy  drivers.  It  discusses  each  strategy  driver  to  assist  program 
personnel  to  assess  their  program’s  software  acquisition  risk. 
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Section  4  discusses  acquisition  strategies  in  more  detail  and  sample  acquisition  strategy  ele¬ 
ments  including  Acquisition  Approach,  Competition,  Solicitation,  Contract  Approach,  Train¬ 
ing,  and  Source  of  Support.  This  is  not  a  comprehensive  set  of  strategy  elements;  it  is  only  a 
subset  used  to  illustrate  the  acquisition  strategy  development  process  proposed  in  this  report. 

Section  5  provides  a  method  for  performing  a  comparative  analysis  of  the  strategy  drivers 
and  the  strategic  choices  for  each  strategy  element.  The  method  uses  a  technique,  slider  bars 
to  support  a  more  systematic  approach  to  reasoning  about  software  risk  on  a  program.  This 
section  describes  how  to  profile  a  program’s  strategy  drivers,  its  acquisition  strategy  elements 
and  corresponding  strategic  choices,  and  the  relationship  between  the  drivers  and  strategic 
choices  for  each  element. 

Appendix  A  provides  a  hard  copy  of  the  template  used  in  the  ASDT.  This  tool  is  discussed 
further  in  Section  1.3.  The  template  can  be  used  to  profile  the  program’s  key  strategy  drivers. 
It  can  also  be  used  to  identify  specific  strategy  element  choices  and  evaluate  how  those 
choices  mitigate  the  program’s  software  risks. 

1.3  Acquisition  Strategy  Development  Tool 

The  purpose  of  the  ASDT  is  to  help  government  program  offices  formulate  and  evaluate  how 
their  acquisition  strategies  address  the  major  software  risks  faced  by  the  program  in  a  more 
systematic  way.  It  can  be  used  to  profile  the  program’s  software  acquisition  characteristics 
and  identify  key  strategy  drivers.  It  can  also  be  used  to  identify  specific  strategic  choices  and 
evaluate  how  those  choices  mitigate  the  program’s  software  risks. 

Acquisition  planners  can  apply  the  method  and  techniques  discussed  in  this  report  without 
using  the  ASDT.  The  ASDT  is  provided  for  acquisition  planners  so  that  each  acquisition  or¬ 
ganization  does  not  have  to  design  templates  from  scratch.  The  ASDT  is  available  for 
download  on  the  SEI  Publications  Web  site  at 
http://www.sei.cmu.edu/publications/publications.html. 

Instructions  on  how  to  use  the  ASDT  are  contained  within  the  workbook.  The  ASDT  requires 
Microsoft  Office  Excel  2003  or  newer.  Macro  execution  must  be  enabled  for  it  to  work  prop¬ 
erly. 
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2  Developing  Acquisition  Strategies 


2.1  Acquisition  Planning  Includes  an  Acquisition 
Strategy  and  an  Acquisition  Plan 

Acquisition  planning  is  defined  as 

The  process  by  which  the  efforts  of  all  personnel  responsible  for  an  acquisition 
are  coordinated  and  integrated  through  a  comprehensive  plan  for  fulfilling  the 
agency  need  in  a  timely  manner  and  at  a  reasonable  cost.  It  is  performed 
throughout  the  life  cycle  and  includes  developing  an  overall  acquisition  strategy > 
for  managing  the  acquisition  and  a  written  Acquisition  Plan  (AP)  [DAU  03]. 

As  shown  in  Figure  3,  acquisition  planning  is  the  act  of  defining  and  maintaining  an  overall 
approach  for  a  program.  Acquisition  planning  guides  all  elements  of  program  execution  to 
transform  the  mission  need  into  a  fielded  system  that  is  fully  supported  and  delivers  the  de¬ 
sired  capability.  The  goal  of  acquisition  planning  is  to  provide  a  roadmap  that  will  be  fol¬ 
lowed  to  maximize  the  chances  of  successfully  fielding  a  system  that  meets  users’  needs 
within  cost  and  schedule.  Acquisition  planning  is  an  iterative  process;  feedback  loops  impact 
future  acquisition  planning  activities. 


Figure  3:  The  Acquisition  Planning  Context 

Software  acquisition  planning  is  a  powerful  process  that  can  help  define  the  most  promising 
acquisition  options  that  best  mitigate  risks  in  the  development  of  software.  Program  risks  take 
many  forms  and  come  from  many  sources.  The  first  challenge  to  the  program  management 
team  is  to  identify  all  of  the  affected  stakeholders.  The  second  challenge  is  to  apply  the  ex¬ 
pertise  of  program  office  staff  and  these  stakeholders  to  identify  and  prioritize  the  program’s 
goals  and  risks.  After  these  two  challenges  are  met,  it  is  possible  to  reason  about  and  create 
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an  acquisition  strategy  that  mitigates  the  highest  priority  risks  and  manages  the  remaining 
risks. 

An  acquisition  strategy ,  when  formulated  carefully,  can  be  a  means  of  addressing  program 
risks  via  program  structure.  More  formally,  an  acquisition  strategy  is 

A  business  and  technical  management  approach  designed  to  achieve  program 
objectives  within  the  resource  constraints  imposed.  It  is  the  framework  for  plan¬ 
ning,  directing,  contracting  for,  and  managing  a  program.  It  provides  a  master 
schedule  for  research,  development,  test,  production,  fielding,  modification, 
postproduction  management,  and  other  activities  essential  for  program  success. 

The  acquisition  strategy >  is  the  basis  for  formulating  functional  plans  and  strate¬ 
gies  (e.g.,  Test  and  Evaluation  Master  Plan  (TEMP),  Acquisition  Plan  (AP), 
competition,  systems  engineering,  etc.)  [DAU  03]. 

The  “best”  acquisition  strategy  for  a  given  program  directly  addresses  that  program’s  highest 
priority  risks.  High-priority  risks  can  be  technical,  if  no  one  has  yet  built  a  component  that 
meets  some  critical  aspect  of  the  system  or  has  never  combined  mature  components  in  the 
way  that  is  required.  Risks  can  be  programmatic  if  the  system  must  be  designed  to  accommo¬ 
date  predefined  cost  or  schedule  constraints.  Or,  risks  can  be  mission-related  when  the  char¬ 
acteristics  of  a  system  that  meets  the  need  cannot  be  fully  articulated  and  agreed  upon  by 
stakeholders.  Each  program  faces  a  unique  set  of  risks,  so  the  corresponding  acquisition  strat¬ 
egy  must  be  unique  to  address  them. 

In  addition,  program  risks  change  over  the  course  of  the  program  as  new  risks  are  discovered 
and  previously  identified  risks  are  mitigated.  The  selected  acquisition  strategy  must  evolve  to 
respond  to  what  is  known  (and  unknown)  about  the  goals,  objectives,  and  constraints  of  the 
system;  the  state  of  the  requirements;  the  maturity  of  the  technology,  including  the  software; 
and  the  available  resources  such  as  budget,  people,  and  skills.  Much  as  a  commander  creates 
and  adjusts  a  course  of  action  based  on  a  sense  of  the  threat  and  the  operational  constraints 
and  resources  available  to  meet  that  threat,  a  program  manager  uses  acquisition  planning  to 
continually  adjust  the  program’s  acquisition  strategy  to  accommodate  the  program’s  risks  as 
they  are  currently  understood. 

Figure  4  represents  a  high-level  graphical  view  of  acquisition  strategy  evolution.  First,  an 
initial  strategy  is  developed  and  executed.  Then,  the  acquisition  strategy  is  refined  based  on 
progress  and  modifying  forces  to  the  program.  This  cycle  is  repeated  multiple  times  until  the 
program  completes. 
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Figure  4:  The  Acquisition  Strategy  Development  Process 

As  stated  previously,  an  acquisition  strategy  is  one  of  two  major  components  of  acquisition 
planning.  The  second  necessary  component  is  a  written  acquisition  plan.  An  acquisition  plan 
is 


a  formal  written  document  reflecting  the  specific  actions  necessaiy  to  execute  the 
approach  established  in  the  approved  acquisition  strategy’  and  guiding  contrac¬ 
tual  implementation  [DAU  03] 


The  terms  acquisition  planning,  acquisition  strategy,  and  acquisition  plan  are  frequently  used 
interchangeably,  which  causes  much  confusion.  A  program’s  acquisition  strategy  is  different 
from  its  acquisition  plan,  but  both  are  artifacts  of  the  process  of  acquisition  planning. 


2.2  Key  Aspects  of  Acquisition  Strategy 
Development 

An  acquisition  strategy  must  take  into  consideration  many  elements  to  accomplish  system 
objectives  during  the  course  of  the  system’s  life  cycle  [DoD  96].  An  acquisition  strategy  in¬ 
cludes  a  collection  of  strategy >  elements.  Each  strategy  element  is  a  decision  or  a  plan  on  how 
to  proceed  with  a  specific  aspect  of  the  program  execution.  Examples  of  strategy  elements 
include,  but  are  not  limited  to,  the  acquisition  approach,  contract  approach,  and  competition. 
Section  4  describes  strategy  elements  in  more  detail. 

The  acquisition  strategy  guides  decisions  made  throughout  the  life  of  the  program.  It  should 
be  based  on  reasoning  about  both  internal  and  external  complex  factors  with  regard  to  the 
system  being  built,  operated,  and  maintained.  When  developing  an  acquisition  strategy,  it  is 
important  to  understand  the  program’s  driving  factors.  Drivers  are  conditions  that  impact 
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program  risk.  For  example,  the  skills  of  the  program  office  personnel,  the  number  of  stake¬ 
holders,  and  the  number  of  system  configurations  that  need  to  be  maintained  are  all  drivers. 
Drivers  can  influence  the  acquisition  strategy  independently  or  there  can  be  interactions  be¬ 
tween  drivers  that  affect  program  risk.  For  example,  unstable  requirements  usually  influence 
overall  costs.  Section  3  contains  a  taxonomy  to  help  acquisition  planners  profile  the  program 
and  determine  the  extent  to  which  it  is  exposed  to  software  risk. 

Figure  5  shows  the  relationships  among  the  program’s  drivers,  its  risks,  and  its  acquisition 
strategy.  The  first  step  is  to  determine  a  program’s  internal  and  external  drivers. 


After  identifying  the  major  drivers,  you  need  to 


•  determine  which  drivers  present  the  highest  risk  to  your  program 

•  determine  how  the  drivers  will  influence  your  program’s  acquisition  strategy  elements 

•  formulate  strategies  (by  making  choices  among  alternatives)  that  you  believe  will  best 
address  the  highest  risk  drivers 

•  analyze  the  strategies  to  determine  gaps  and  remaining  high  risk  areas  and  determine  if 
these  risks  can  be  mitigated  through  other  options 

No  strategy  can  mitigate  all  of  a  program’s  risk;  identify  the  strategy  that  provides  the  best 
risk  mitigation  and  can  be  executed  by  the  available  staff  and  within  the  available  budget. 
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Figure  5:  Risk  Management  Via  Acquisition  Strategy 
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2.3  Software  Risk  in  Acquisition  Planning 

Acquisition  planning  must  address  the  risks  that  can  arise  during  development,  use,  and 
maintenance  of  software.  To  this  end,  the  acquisition  strategy  must  reflect 

•  the  degree  to  which  software  determines  the  capabilities  and  qualities  of  each  system 
component  or  sub-component 

•  all  software  elements  that  are  potentially  difficult  or  costly  to  implement 

•  the  operational,  maintenance,  and  sustainment  implications  of  software  components 

•  the  threat  that  faulty  or  failed  software  poses  to  human  life,  safety,  security,  environment, 
and  so  on 

Some  acquisition  planners  neglect  to  address  the  unique  nature  of  software  risks.  They  ignore 
them  or  treat  them  as  an  insignificant  part  of  the  system  implementation  risks  and  assume 
that  they  will  be  managed  by  the  implementation  contractor.  This  paradigm  persists  for  a  va¬ 
riety  of  reasons. 

•  The  acquisition  community  as  a  whole  does  not  address  software  early  enough  in  system 
development.  Therefore,  many  critical  acquisition  planning  decisions  are  made  with  scant 
consideration  of  software. 

•  Acquisition  planners  have  no  empirical  foundation  for  selecting  one  acquisition  approach 
over  another.  This  makes  it  difficult  to  know  whether  a  given  approach  can  successfully 
mitigate  software  risks  and  encourages  managers  to  fall  back  on  the  approaches  with 
which  they  are  most  familiar;  these  approaches  usually  ignore  software  risks. 

2.3.1  The  Challenge  of  Software 

Increasingly,  software  is  the  major  determinant  of  system  functionality.  In  the  year  2000,  the 
Defense  Science  Board  found  tremendous  growth  in  the  software  content  of  both  manned  and 
unmanned  systems  [DSB  00].  In  fact,  software  requirements  now  represent  the  bulk  of  over¬ 
all  specification  requirements  for  most  systems.  For  example,  80%  of  the  U.S.  Air  Force’s 
requirements  for  building  the  F/A-22  Raptor  and  65%  of  the  requirements  for  the  B-2  Spirit 
Stealth  Bomber  were  software  requirements  [Nelson  99]. 

Within  the  U.S.  Army,  the  growing  dependence  on  software  is  equally  astounding.  The  Ml 
Abrams  Main  Battle  Tank  (MBT)  performs  its  mission  with  the  help  of  approximately 
600,000  (0.6  M)  source  lines  of  software  code  (SLOC)  [Nelson  99],  Planned  Army  systems 
such  as  those  under  development  by  the  Future  Combat  Systems  network  employ  far  more 
software  (34M  SLOC)  [GAO  05]. 
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Figure  6:  A  Survey  of  Software  Projects  According  to  the  Stand ish  Group 

Due  to  the  advanced  capabilities  now  required,  software  increasingly  drives  the  cost,  sched¬ 
ule,  and  quality  of  integrated,  hybrid  (software  and  hardware)  systems.  As  shown  in  Figure  6, 
the  percentage  of  software  projects  completed  on  time  seems  to  be  improving.  Flowever,  in 
2002,  51%  of  software  projects  were  late,  over  budget,  and/or  lacked  critical  features — and 
this  number  excludes  an  additional  15%  of  projects  that  were  cancelled.  In  addition,  the  aver¬ 
age  final  software  product  contains  only  52%  of  the  originally  specified  features  [Standish 
03]. 

Systemic  analysis  of  results  from  the  Tri-Services  Assessment  Initiative  suggest  that  there 
has  been  “little  positive  impact  from  past  corrective  actions,  initiatives,  and  policy”  in  the 
Department  of  Defense  (DoD)  [McGarry  03].  There  are  many  reasons  why  software,  particu¬ 
larly  DoD  software,  has  proven  so  difficult  to  build.  We  cannot  possibly  list  them  all  in  this 
document,  but  we  can  reflect  on  several  reasons  that  illustrate  why  formulating  an  acquisition 
strategy  is  critical. 

1 .  It  is  likely  that  no  organization  acquires  a  wider  range  of  software  than  the  DoD.  In  the 
Army  alone,  software  ranges  from  well-defined  business  systems  to  systems  supporting 
the  acquiring,  tracking,  and  destroying  of  a  missile  moving  faster  than  a  bullet.  The 
broad  range  of  software  required  to  support  these  two  different  missions  implies  that  an 
equally  broad  range  of  strategies,  processes,  and  technologies  must  be  employed. 

2.  Some  of  the  most  complex  types  of  software  produced  by  the  DoD  are  components  em¬ 
bedded  inside  large  combat  systems.  In  many  of  these  systems,  software  is  an  enabler 
that,  when  used  in  combination  with  other  technologies,  can  be  used  to  address  sys- 
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tem/subsystem  requirements.  In  many  cases,  alternative  technologies  could  be  used  to 
provide  the  same  functionality — albeit  with  different  implications.  The  complexity  of 
the  software  required  is  rarely  evident  when  the  system  is  conceived  and  the  software 
requirements  tend  to  grow  in  complexity  as  hardware  component  builders  make  assump¬ 
tions  about  specific  systems  and  the  community  of  interacting  systems.  Eventually, 
software  must  be  developed  to  fill  the  gaps  and  to  integrate  individual,  custom  systems. 

3.  Engineers  in  other  disciplines  understand  the  limitations  of  the  materials  and  associated 
fabrication  techniques  within  their  discipline.  For  example,  there  are  rating  scales  for 
properties  of  metals  such  as  thermal  stability,  tensile  strength,  hardness,  elasticity,  and  so 
on.  On  the  other  hand,  there  are  few,  if  any,  hard  and  fast  “rules”  of  software  engineer¬ 
ing.  In  this  environment,  planners,  developers,  stakeholders,  and  users  all  tend  to  make 
optimistic  assumptions  about  software — that  it  is  infinitely  malleable  and  can  be  made  to 
exhibit  unprecedented  degrees  of  interesting  characteristics  (e.g.,  dependability,  per¬ 
formance,  and  security).  In  effect,  flexibility  is  the  greatest  asset  of  software  but  is  also 
its  key  liability. 

4.  The  DoD  acquisition  community  is  rightly  focused  on  acquiring  capabilities  to  support 
the  warfighter.  However,  this  tight  focus  sometimes  leads  to  reduced  emphasis  on  the 
problems  of  developing  appropriate  software  to  support  these  capabilities  because  the 
engineering  of  the  software  becomes  secondary  to  the  engineering  of  the  hardware  for 
the  system.  Because  software  is  inherently  intangible  and  often  a  relatively  small  part  of 
the  system,  other  system  issues  overwhelm  it.  As  a  result,  software  development  is  not 
properly  planned,  started  late,  defined  late,  under-budgeted,  and  so  on.  Ultimately,  a 
software  implementation  that  might  have  been  straightforward  at  one  time  stagnates  un¬ 
til  there  is  no  option  but  to  attempt  to  develop  a  now-complex  software  component  on  a 
shortened  schedule. 

Effective  software  acquisition  can  be  a  point  of  great  leverage.  Addressing  even  some  of  the 
problems  with  software  in  major  system  acquisitions  could  lead  to  huge  savings  and  better 
systems.  The  DoD  has  recognized  this  for  many  years  and  has  attempted  to  improve  the 
situation  by  instituting  various  standards  such  as  DoD-STD-2167A,  MIL-STD-498, 

IEEE/EIA  1207,  by  embracing  process  improvement  methodologies  such  as  ISO  9000  and 
the  Capability  Maturity  Model®  Integration  (CMMI®)  framework,  and  by  adopting  a  wide 
range  of  newer,  more  flexible  technologies  such  as  Extensible  Markup  Language  (XML). 
However,  in  almost  all  cases,  as  evidenced  by  the  Tri-Services  Assessment  Initiative,  results 
have  been  disappointing  [McGarry  03]. 

2.3.2  Planning  to  Mitigate  Software  Risk 

Software  acquisition  planning  is  a  powerful  process  that  can  help  define  the  most  promising 
acquisition  options  that  best  mitigate  risks  in  the  development  of  software.  An  acquisition 
strategy  should  clearly  describe  the  program’s  approach  for  defining,  building,  fielding,  and 
supporting  the  system  by 
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•  documenting  what  the  program  understands  to  be  the  need  and  the  environmental  con¬ 
straints  under  which  the  system  must  be  built,  operated,  and  maintained 

•  defining  the  means  by  which  requirements  that  reflect  the  need  will  be  developed  and 
managed 

•  establishing  appropriate  relationships  and  commitments  among  stakeholders,  program 
offices,  and  contractors  for  the  life  of  the  system 

•  providing  a  framework  under  which  a  contract  (or  contracts)  is  managed  to  deliver  prod¬ 
ucts  or  services  needed  by  the  program 

•  establishing  mechanisms  to  effectively  communicate  the  approach  that  is  used  by  the 
program  to  all  affected  stakeholders 

A  software  acquisition  strategy  must  effectively  communicate  a  complete,  coherent,  and  con¬ 
sistent  approach  that  is  used  by  the  program  so  that  the  program  can  be  resourced,  staffed, 
and  tracked.  Above  all,  the  software  acquisition  strategy  must  support  the  DOD  directive  to 

acquire  quality  products  that  satisfy  user  needs  with  measurable  improvements 
to  mission  capability  and  operational  support,  in  a  timely  manner,  and  at  a  fair 
and  reasonable  price  [DoD  03  a] 

Software  acquisition  planning  aimed  at  mitigating  software  risk  begins  as  soon  as  it  becomes 
apparent  that  a  materiel  solution  employing  software  might  be  required — or  in  the  earliest 
days  of  Concept  Refinement  for  a  program  that  uses  all  of  the  acquisition  phases  described  in 
DoD  Instruction  Operation  of  the  Defense  Acquisition  System  (DoDI  5000.2)  [DoD  03b].  At 
no  other  point  is  the  program  as  pliable.  A  software  acquisition  strategy  is  custom-built  to 
help  the  system  stakeholders,  program  management  office 1  (PMO),  and  contractors  achieve 
the  best  software  system  possible  within  program  constraints.  While  these  early  steps  in  ac¬ 
quisition  planning  are  the  first  and  arguably  the  foremost  mechanism  available  to  achieve 
objectives  in  the  acquisition  and  sustainment  of  systems,  the  acquisition  strategy  must  con¬ 
tinue  to  evolve  as  concepts  are  refined,  as  technology  is  developed,  as  systems  are  acquired, 
and  as  systems  are  fielded  and  supported — until  the  systems  are  retired. 


The  term  program  management  office  (PMO)  is  used  primarily  by  civil  agencies  and  the  U.S. 
Army  to  refer  to  an  organizational  unit  created  to  centralize  and  coordinate  the  management  of 
projects  under  its  domain.  Note  that  different  government  organizations  use  different  terms  to  re¬ 
fer  to  this  organizational  unit.  For  example,  the  Air  Force  uses  the  term  “system  program  office,” 
or  “SPO,”  while  the  Navy  refers  to  it  as  a  “project  management  office.  In  the  interest  of  simplicity 
and  readability,  the  authors  of  this  report  use  the  term  “PMO”  in  a  general  way. 


12 


CMU/SEI-2006-TR-002 


3  Profiling  Program  Software 
Characteristics  and  Acquisition  Strategy 
Drivers 


This  section  introduces  a  taxonomy  of  categories  and  drivers,  depicted  in  Figure  7,  to  help  a 
program  determine  the  extent  to  which  it  is  exposed  to  software  risk.  The  categories  and  cor¬ 
responding  drivers  are  designed  to  be  used  by  acquisition  planners  to  profile  software  risk 
early  in  the  program.  The  software  risk  profile  should  be  updated  throughout  the  execution  of 
the  program  and  the  acquisition  strategy  should  be  adjusted  to  address  newly  discovered  risks 
and  risks  that  have  been  mitigated. 


Software 

Criticality 

Category 

Acquisition 

Environment 

Category 

Programmatic 

Category 

Organizational 

Category 

Life  Cycle 
Category 

Software 

Criticality 

Policies  and 

Mandates 

Mission  Needs 
and  Scope 
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Capability 

Product 

Definition  and 
Specification 

Supplier  Avail¬ 
ability 

Funding 

Stakeholders 

Architecture 
and  Design 

Schedule 

Supplier 
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Verification 
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Deployment 

Maintenance 
and  Support 

Disposal 

Figure  7;  Categories  of  Program  Software  Characteristics  and  Subseguent 
Strategy  Drivers 
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As  shown  in  Figure  7,  program  characteristics  and  strategy  drivers  are  aggregated  into  five 
main  categories: 

1.  Software  criticality  quantifies  the  extent  to  which  the  system  is  “software-bound.”  Sys¬ 
tems  are  considered  software-bound  when  they  require  large  amounts  of  software,  highly 
complex  software,  or  software  that  is  directly  responsible  for  fulfilling  critical  aspects  of 
the  system  mission. 

2.  Acquisition  environment  program  characteristics  and  associated  strategy  drivers  quantify 
the  extent  to  which  the  environment  in  which  the  acquisition  occurs  affects  the  software 
risk  in  the  program.  The  level  of  acquisition  environment  risk  can  be  represented  in 
terms  of  policies  and  mandates  imposed  on  the  program  and  the  availability  of  suppliers. 

3.  Programmatic  program  characteristics  and  associated  strategy  drivers  quantify  the  ex¬ 
tent  to  which  the  fundamental  programmatic  aspects  of  scope,  funding,  and  schedule  af¬ 
fect  the  software  risk  in  the  program. 

4.  Organizational  program  characteristics  and  associated  strategy  drivers  quantify  the  ex¬ 
tent  to  which  organizational  factors  inside  and  outside  the  program  affect  the  software 
risk  in  the  program.  The  level  of  organizational  software  risk  can  be  represented  in  terms 
of  the  characteristics  of  the  program  management  office,  supplier,  and  stakeholders. 

5.  Life-cycle  program  characteristics  and  associated  strategy  drivers  quantify  the  extent  to 
which  various  life  cycle  facets  pose  risk  to  the  program.  The  level  of  life-cycle  software 
risk  can  be  represented  in  terms  of  the  risks  associated  with  the  definition,  specification, 
architecture,  design,  implementation,  test,  deployment,  operations  and  maintenance,  and 
disposal  of  the  system. 

Each  of  these  categories,  subsequent  strategy  drivers,  and  sublevel  drivers  is  discussed  in 
more  detail  in  the  following  sections.  One  or  more  of  the  drivers  may  be  difficult  to  deter¬ 
mine  or  not  applicable  during  a  particular  program  time  frame. 

3.1  Software  Criticality  Category 

3.1.1  Software  Criticality  Driver 

The  extent  to  which  an  individual  program  must  specifically  focus  on  and  address  software 
risks  depends  on  the  criticality  of  software  within  the  capability  to  be  deployed.  To  determine 
software  criticality  proportion  and  reliance  need  to  be  evaluated. 

Proportion  is  the  ratio  of  the  software  to  the  entire  system  and  is  usually  evaluated  using  life- 
cycle  cost,  time  needed  to  produce  and  maintain  the  software,  and  functionality  provided.  For 
example,  is  software  the  majority  of  the  system  or  a  minor  part  of  the  system?  For  certain 
categories  of  business  systems,  such  as  financial,  human  resources,  inventory  management 
systems,  and  for  other  categories  of  systems  that  support  the  mission  of  the  DoD  such  as 
Command,  Control,  Communications,  Computers,  Intelligence  and  Reconnaissance  systems, 
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the  proportion  of  the  software  to  the  system  is  obvious.  For  these  types  of  systems,  software 
virtually  is  the  system. 


For  other  types  of  systems,  however,  the  significance  of  software  components  is  not  as  obvi¬ 
ous.  In  these  systems,  the  software  may  not  be  configured  as  a  set  of  large  applications  that 
appear  on  a  software  architecture  diagram  at  the  system  level,  but  instead,  software  exists  in 
the  system  as  relatively  small  chunks  distributed  across  a  number  of  other  systems  or  devices. 
In  this  case,  the  relationship  between  the  software  components  and  hardware  components 
may  be  more  dominant  than  the  relationship  between  the  individual  pieces  of  software.  The 
proportion  of  software  might  be  small,  but  the  degree  to  which  the  important  functionality  of 
the  system  depends  on  software  is  large.  That  is  why  an  evaluation  of  reliance  is  necessary  to 
determine  software  criticality. 

Reliance  is  the  degree  to  which  the  important  functionality  of  the  system  depends  on  the 
software.  For  example,  a  system  running  flight  control  software  that  is  needed  to  fly  a  heli¬ 
copter  has  a  higher  reliance  on  software  than  a  system  that  employs  software  as  a  means  of 
modeling  certain  system  characteristics. 

What  can  be  deceiving  about  the  criticality  of  the  software  to  be  procured,  however,  is  that  in 
many  modem  systems,  evaluating  proportion  alone  is  insufficient  and  the  reliance  the  system 
has  on  the  software  must  also  be  evaluated.  For  example,  modern  cars — like  their  1909 
Model  T  counterparts — are  systems  that  rely  on  a  drive  train  to  impel  the  wheels,  a  steering 
linkage  to  control  the  direction  of  travel,  and  brakes  to  stop  them.  Unlike  the  Model  T,  how¬ 
ever,  virtually  no  modem  car  can  perform  in  more  than  a  “limp  home”  mode  without  soft¬ 
ware  to  control  many  mechanical  functions. 

The  software  used  by  the  2003  BMW  745Li,  shown  in  Figure  8,  is  a  good  illustration  of  the 
increasing  software  control  of  hardware  systems  in  automobiles.  Among  the  many  features 
that  are  likely  to  rely  on  software  are  the  “mayday”  phone,  on-board  navigation  system,  dy¬ 
namic  stability  control,  dynamic  traction  control,  active  roll  stabilization,  dynamic  brake  con¬ 
trol,  coded  drive-away  protection,  an  adaptive  automatic  transmission,  and  iDrive  systems. 
This  list  can  be  extended  to  include  the  software  that  controls  engine  performance,  emissions, 
and  fuel  consumption  found  on  virtually  all  modem  cars. 
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Figure  8:  BMW  745Li  Software 

The  trend  toward  increased  software  control  can  also  be  seen  in  U.S.  military  systems.  For 
example,  in  the  1975  version  of  the  F- 15  Eagle  only  35%  of  functions  required  software  sup¬ 
port.  That  number  grew  to  80%  for  the  F/A-22  Raptor  introduced  20  years  later  [Nelson  99]. 


One  definition  of  the  criticality  of  the  software  in  a  system  can  be  found  in  Table  1. 


Table  1:  Relationships  Among  Criticality,  Reliance,  and  Proportion 


Criticality 

Reliance 

Proportion 

Very  High 

Complete  reliance  on  software 

Majority  of  the  system 

High 

High  reliance  on  software 

Majority  of  the  System 

Medium 

High  reliance  on  software 

Minor  part  of  the  system 

Low 

Low  reliance  on  software 

Minor  part  of  the  system 

Very  Low 

No  reliance  on  software 

Software  is  not  part  of  the  system 

The  software  in  a  system  (or  software  used  to  model,  simulate,  or  test  a  system)  cannot  be 
categorized  simply  as  critical  or  not.  Most  programs  categorize  the  software  somewhere  be¬ 
tween  the  two  extremes.  The  extent  of  the  software  criticality  can  be  rated  on  a  sliding  scale 
that  extends  from  one  extreme  to  the  other:  Very  Low  to  Very  High. 

If  software  criticality  is  low,  software  risks  are  not  dominant  factors  in  acquisition  planning. 
However,  systems  that  are  not  software-critical  are  increasingly  rare.  Therefore,  most  pro¬ 
grams  today  need  a  way  to  reason  about  software  risks  and  the  potential  for  the  chosen  acqui¬ 
sition  strategy  to  mitigate  software  risk  within  the  system. 
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3.2  Acquisition  Environment  Category 

Evaluating  acquisition  environment  program  characteristics  and  associated  strategy  drivers, 
including  policies  and  mandates  and  supplier  availability,  can  help  quantify  the  type  and 
amount  of  software  risk  in  the  program. 

3.2.1  Policies  and  Mandates  Driver 

Policies  and  mandates  are  external  constraints  that  a  higher  authority  imposes  on  a  program. 
One  definition  of  policy  is  “a  definite  course  or  method  of  action  selected  from  among  alter¬ 
natives  and  in  light  of  given  conditions  to  guide  and  determine  present  and  future  decisions” 
and  another  is  “a  high-level  overall  plan  embracing  the  general  goals  and  acceptable  proce¬ 
dures  especially  of  a  governmental  body.”  The  dictionary  also  defines  a  mandate  as  “an  au¬ 
thoritative  command”  that  a  higher  authority  imposes  [Merriam-Webster  05]. 

Policies  and  mandates  come  in  several  forms  and  can  be  described  by  the  examples  shown  in 
Table  2. 


Table  2:  Examples  of  Policies  and  Mandates 


Policy  or  Mandate 

Example 

Compliance  with  one  or  more  laws 

•  OSHA 

•  Privacy  Act 

Compliance  with  one  or  more  directives  or 

instructions 

•  Command,  Control,  Communications,  Computers,  Intelli¬ 
gence,  Surveillance  and  Reconnaissance  (C4ISR) 

•  Business  Enterprise  Architecture  (BEA) 

•  Joint  Technical  Architecture  (JTA) 

•  DoD  8500  series  on  Infonnation  Assurance 

Compliance  with  a  specific  practice 

•  Use  of  perfonnance  requirements  instead  of  detailed 

specifications 

•  Use  of  a  Statement  of  Objective  (SOO)  instead  of  a 

Statement  of  Work  (SOW)  in  the  initial  solicitation 

•  Use  of  a  Capabilities  Development  Document  (CDD) 

instead  of  an  Operational  Requirements  Document  (ORD) 

Compliance  with  specific  contracting  elements 

•  8A  requirements 

Required  certifications  and  accreditations 

•  Certificate  of  Networthiness  (CoN) 

•  Certificate  to  Operation  (CtO) 

Use  of  Government  Furnished  Infonnation  (GFI) 

and  Equipment  (GFE) 

•  Incorporation  of  a  specific  component 

•  Providing  the  contractor  with  a  development  and  testing 

environment 
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When  formulating  an  acquisition  strategy,  acquisition  planners  must  consider  the  amount  of 
conflict  between  the  policies  and  mandates  imposed  on  a  program,  the  relative  stability  or 
instability  of  a  particular  policy  or  mandate,  the  degree  to  which  policies  and  mandates  con¬ 
flict  with  program  needs,  and  the  amount  of  control  a  program  has  to  modify  a  policy  or 
mandate. 

Analyzing  the  applicability  of  mandates,  resolving  any  conflicts,  and  determining  their  im¬ 
pact,  regardless  of  their  relevance  to  the  program,  requires  resources.  Traversing  the  paper 
trail  to  verify  compliance  with  policies  and  mandates  can  be  quite  time-consuming;  these 
tasks  should  not  be  underestimated  during  planning.  Information  about  regulations  abounds, 
but  gathering  information  about  every  single  policy  and  mandate  that  applies  to  a  program  is 
a  nontrivial  task.  For  example 

•  Who  is  responsible  for  figuring  out  the  detail  of  compliance:  the  PMO  or  the  contractor? 

•  What  is  the  risk  something  will  be  overlooked? 

•  What  is  the  risk  that  a  policy  or  mandate  can  be  interpreted  in  multiple  ways? 

The  more  policies  and  mandates  imposed  on  a  program,  the  more  resources  consumed,  and 
more  likely  that  one  policy  or  mandate  will  conflict  with  another  policy  or  mandate.  Also, 
after  a  conflict  between  mandates  is  identified,  there  may  or  may  not  be  a  consistent  mecha¬ 
nism  to  resolve  the  conflict  and  programs  confronted  with  such  conflicts  rarely  receive  guid¬ 
ance  on  how  to  resolve  them.  The  PMO  may  assume  some  risk  when  the  contractor  resolves 
a  conflict  that  the  PMO  doesn’t  know  about  or  if  the  PMO  does  not  fully  understand  what  the 
resolution  means.  Whether  the  PMO  or  contractor  attempts  to  resolve  a  conflict  between 
mandates,  there  is  a  risk  that  the  resolution  can  cause  technical  incompatibilities  with  other 
systems  in  the  future  or  inefficiencies  in  the  current  system. 

In  addition  to  conflicts  arising  between  individual  policies  and  mandates,  conflicts  between 
the  policies  and  mandates  and  the  needs  of  the  program  can  also  arise.  These  types  of  con¬ 
flicts  can  be  manifested  in  many  forms  including 

•  expense  of  conforming  to  policies  and  mandates  without  having  additional  funding  to  do 
so.  For  example,  a  requirement  to  use  a  specific  software  component  might  be  imposed, 
but  funding  for  the  sustainment  of  that  component  is  not  provided  or  guaranteed. 

•  requirements  that  specific  practices  be  followed  (or  artifacts  created)  that  are  untried  and 
unproven  and  have  not  been  adequately  piloted  before  being  imposed.  For  example,  one 
program  producing  real-time  embedded  software  was  directed  to  develop  all  code  in  Java 
at  a  time  when  Java  did  not  support  real-time  software. 

•  expense  of  significant  rework  and  schedule  degradation  when  policies  or  mandates  are 
levied  changes  after  the  system  has  already  been  designed 

•  requirements  to  use  a  solution  developed  by,  or  for,  the  DoD  that  competes  with  more 
viable  commercial  alternatives 
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•  schedule  disruption  caused  by  mandates.  For  example,  meeting  National  Security  Agency 
certification  requirements  can  become  a  critical  path  in  the  program  schedule  and  must 
be  planned. 

•  work  to  determine  which  policies  and  mandates  apply  to  your  program  and  what  the  im¬ 
pacts  are;  in  a  large  program,  determining  how  the  policies  and  mandates  can  be  applied 
consistently  to  all  facets  of  the  program.  Even  allocating  the  proper  staff  to  monitor  these 
issues  can  be  difficult  for  a  PMO. 

As  in  the  case  of  conflicts  between  policies  and  mandates,  after  a  conflict  is  identified  be¬ 
tween  policies  and  mandates  and  program  needs,  there  may  or  may  not  be  a  consistent 
mechanism  to  resolve  the  conflict.  In  addition,  alternative  solutions  may  not  be  allowed. 

The  stability  or  instability  of  a  policy  or  mandate  must  also  be  considered.  Is  there  a  risk  that 
the  program  will  adhere  to  a  policy  or  mandate  that  will  change  after  key  program  decisions 
have  been  made?  What  is  the  likelihood  that  a  policy  or  mandate  will  be  introduced  midway 
through  the  program  that  would  require  the  solution  to  be  reworked? 

A  program’s  ability  to  resolve  policy  and  mandate  conflicts  may  counterbalance  the  risk  im¬ 
posed  by  the  number  and  extent  of  the  conflicts.  It  is  important  to  have  a  good  understanding 
of  which  conflicts  are  beyond  PMO  resolution  so  that  appropriate  higher  level  help  can  be 
sought.  The  presence  of  too  many  irresolvable  conflicts  can  be  a  warning  sign  that  the  pro¬ 
gram  cannot  be  executed  as  currently  planned. 

The  extent  to  which  a  program  has  policy  and  mandate  conflicts  can  be  rated  on  a  sliding 
scale  that  extends  from  one  extreme  to  the  other:  Low  to  Fligh. 

3.2.2  Supplier  Availability  Driver 

Another  external  enviromnental  factor  to  consider  when  evaluating  an  acquisition  strategy  is 
the  number  of  available,  qualified  suppliers  of  required  software  components.  Flow  many 
suppliers  can  potentially  fulfill  the  needs  of  the  program?  Is  the  program  constrained  by  a 
limited  set  of  pre-qualified  suppliers?  Does  their  expertise  match  the  needs  of  the  program? 
Are  there  unique  aspects  of  the  program  that  can  only  be  supported  by  a  limited  number  of 
suppliers?  Do  the  suppliers  have  adequate  capacity  to  execute  all  of  the  expected  work  within 
the  required  time  frame?  The  number  of  available,  qualified  suppliers  can  be  rated  on  a  slid¬ 
ing  scale  that  extends  from  one  extreme  to  the  other:  Low  to  Fligh. 

3.3  Programmatic  Category 

Programmatic  program  characteristics  quantify  the  extent  to  which  fundamental  program¬ 
matic  strategy  drivers  of  mission  needs  and  scope,  funding,  and  schedule  affect  the  type  and 
amount  of  software  risk  in  the  program. 
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3.3.1  Mission  Needs  and  Scope  Driver 

Systems  are  developed  to  fulfill  the  needs  of  the  mission.  Every  acquisition  plans  to  fulfill  all 
defined  mission  needs  and  deliver  a  product  that  satisfies  100%  of  the  user  requirements. 

And  yet  we  see  from  the  Standish  Group’s  CHAOS  report  that  “successful”  systems  meet 
only  52%  of  requirements  [Standish  03].  Clearly,  the  mission  needs  contain  some  degree  of 
flexibility.  The  flexibility  of  the  mission  that  the  system  needs  to  fulfill  is  a  strategy  driver. 
For  some  missions,  the  scope  of  the  mission  is  very  exact  and  constricted.  For  others,  it  is 
much  more  flexible.  The  extent  to  which  the  mission  needs  are  constrained  can  be  rated  on  a 
sliding  scale  that  extends  from  one  extreme  to  the  other:  Flexible  to  Rigid. 

3.3.2  Funding  Drivers 

The  funding  drivers  described  in  this  section  include  funding  constraints  and  the  funding  pro¬ 
file. 

3.3.2. 1  Funding  Constraints 

Funding  constraints  determine  the  degree  to  which  the  amount  of  funding  allocated  to  the 
program  matches  the  estimated  funding  required  to  develop,  operate,  and  maintain  the  sys¬ 
tem.  Funding  constraints  can  significantly  impact  both  “software-critical”  and  “non-software- 
critical”  systems.  Inadequate  funding  can  cause  suboptimal  engineering  choices  to  be  made  in 
the  interest  of  saving  money;  choices  such  as  deferring  functionality  or  allocating  require¬ 
ments  to  software  that  might  be  better  implemented  in  hardware. 

How  does  software  influence  the  risk  associated  with  funding  constraints?  A  key  considera¬ 
tion  is  the  life-cycle  cost  and  total  cost  of  ownership  estimates  for  the  software  in  the  system. 
According  to  DoDl  5000.2,  sound  cost  estimates  are  based  on  well-defined  programs.  With 
large-scale  unprecedented  systems,  this  becomes  a  difficult  task. 

The  inability  to  develop  credible  and  accurate  cost  estimates  correlates  with  the  level  of  un¬ 
certainty  in  the  program  or  system  definition,  technical  performance,  and  cost  estimating 
method.  In  turn,  this  influences  the  accuracy  of  the  anticipated  funding  profile.  In  contrast, 
the  funding  profile  is  sometimes  dictated  by  the  congressional  budget.  The  program  must 
then  use  its  cost  estimating  ability  to  determine  an  affordable  set  of  requirements. 

Factors  influencing  cost  estimating  include  immature  software  cost  estimation  models,  lack 
of  experience  in  using  cost  models,  lack  of  knowledge  and  experience  in  planning  and  man¬ 
aging  software  development  programs,  lack  of  subject  matter  experts,  immature  and  volatile 
requirements,  the  quality  of  historical  data  from  similar  projects,  and  influences  by  execu¬ 
tives  having  the  authority  to  reject  or  modify  valid  estimates  due  to  “political”  or  other  rea¬ 
sons. 

The  extent  to  which  the  funding  constraints  are  a  fact  of  the  acquisition  can  be  rated  on  a  slid¬ 
ing  scale  that  extends  from  one  extreme  to  the  other:  Few  to  Many. 
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3. 3.2.2  Funding  Profile 

The  type  and  timing  of  funding  dollars  applied  to  the  acquisition  can  also  drive  decisions 
made  about  the  acquisition  strategy.  It  is  not  just  how  much  funding  the  program  receives, 
but  also  the  type  of  appropriation  and  when  it  is  received.  For  example,  a  program  that  needs 
to  do  a  lot  for  research  and  development  and  will  acquire  a  limited  amount  of  hardware  usu¬ 
ally  has  a  front- loaded  budget,  in  which  a  majority  of  the  overall  funding  is  allocated  to  do 
the  work  at  the  beginning  of  the  project  versus  the  end.  In  addition,  this  program  would  need 
research  and  development  dollars. 

In  order  to  develop  high  levels  of  product  quality,  investment  in  up-front  engineering  tasks  is 
necessary.  Tasks  such  as  requirements  definition  and  architecture  drive  the  scope  of  the  soft¬ 
ware  development  effort,  so  inadequate  attention  to  those  tasks  invariably  leads  to  increases 
in  software  size  and  complexity  and  ultimately  to  schedule  delays,  cost  increases,  and  product 
quality  issues.  Consequently,  early  engineering  tasks  need  to  be  funded  adequately  at  the  out¬ 
set  of  the  program. 

Using  an  evolutionary  acquisition  approach  such  as  incremental  or  spiral  can  complicate 
funding  needs.  For  example,  the  program  may  need  to  budget  funding  from  several  different 
appropriation  types  simultaneously. 

The  extent  to  which  the  funding  profile  matches  the  needs  of  the  acquisition  can  be  rated  on  a 
sliding  scale  that  extends  from  one  extreme  to  the  other:  Mismatched  to  Matched. 

3.3.3  Schedule  Drivers 

The  schedule  drivers  described  in  this  section  include  schedule  constraints  and  urgency. 

3.3.3. 1  Schedule  Constraints 

Like  funding  constraints,  schedule  constraints  are  the  degree  to  which  the  schedule  allocated 
to  the  program  matches  the  estimated  time  required  to  develop,  operate,  and  maintain  the  sys¬ 
tem.  Schedule  constraints  can  significantly  impact  both  “software-critical”  and  “non¬ 
software-critical”  systems. 

How  does  software  influence  the  schedule  for  the  system?  What  aspects  of  the  program  have 
key  dependencies  on  certain  software  events?  If  software  is  on  the  critical  path,  software  risks 
need  to  have  very  good  mitigation  plans  and  must  be  constantly  monitored. 

The  ability  to  develop  credible  and  accurate  schedule  estimates  is  influenced  by  the  level  of 
uncertainty  in  the  program  or  system  definition  and  the  schedule  estimating  method  used. 
Other  factors  that  influence  schedule  estimating  include  immature  estimation  models,  lack  of 
experience  in  using  the  models,  lack  of  knowledge  and  experience  in  planning  and  managing 
software  development  programs,  lack  of  subject  matter  experts,  immature  and  volatile  re¬ 
quirements,  the  experience  of  the  personnel  performing  the  estimates  and  the  quality  of  his- 
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torical  information  from  similar  projects,  and  influences  by  executives  having  the  authority 
to  reject  or  modify  valid  estimates  due  to  “political”  or  other  reasons. 

Another  consideration  that  can  have  a  large  impact  on  schedule  risk  is  the  need  to  interface 
with  other  programs.  Risk  is  increased  if,  due  to  interoperability  goals,  the  program  schedule 
must  be  coordinated  with  one  or  more  other  programs. 

The  extent  to  which  schedule  constraints  are  a  fact  of  the  acquisition  can  be  represented  as  a 
continuum  that  extends  from  one  extreme  to  the  other:  Few  to  Many. 

3. 3. 3.2  Urgency 

In  certain  circumstances,  urgency  of  deploying  a  capability  to  the  field  might  have  significant 
impact  on  the  acquisition  strategy.  For  example,  the  current  war  situation  might  be  placing 
pressure  on  certain  acquisitions  to  deliver  capabilities  at  a  different  rate,  in  a  different  se¬ 
quence,  or  with  different  capabilities  than  planned.  How  does  your  acquisition  strategy  ac¬ 
commodate  urgent  needs?  The  level  of  urgency  can  be  rated  on  a  sliding  scale  that  extends 
from  one  extreme  to  the  other:  Low  to  High. 

3.4  Organizational  Category 

Organizational  program  characteristics  and  associated  strategy  drivers  quantify  the  extent  to 
which  organizational  factors  inside  and  outside  the  program  affect  the  software  risk  in  the 
program.  Strategy  drivers  include  PMO  capabilities,  stakeholders,  policies/mandates,  and 
performance  team/contractor  capabilities. 

3.4.1  PMO  Capabilities  Drivers 

The  drivers  pertaining  to  PMO  capabilities  described  in  this  section  include  staff  skills,  ca¬ 
pacity  and  stability,  and  the  process  focus  of  the  PMO  and  its  governing  organization. 

3.4.1. 1  PMO  Staff  Skills 

The  PM  [program  manager]  must  assemble  the  proper  acquisition  strategy  selec¬ 
tion  and  development  team.  It  is  important  to  staff  this  team  with  individuals 
whose  knowledge ,  experience  and  access  to  pertinent  information  equips  them  to 
effectively  address  all  of  the  topics.  The  success  of  each  of  the  succeeding  steps 
in  the  selection  process  depends  on  the  active  participation  of  all  the  members  of 
the  team.  Good  contracting ,  technical  and  business/financial  managers  will  be 
key  payers  in  the  selection  of  the  acquisition  strategy >  [Department  of  the  Navy 
01]. 

With  respect  to  having  the  proper  software  expertise,  the  GAO  noted:  “The  more  managers 
know  about  software  development  processes  and  metrics,  the  better  equipped  they  are  to  ac- 
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quire  software”  [GAO  04a].  In  effect,  the  government  program  manager  needs  to  be  a  “smart 
buyer”  of  the  software  within  the  system. 

While  some  level  of  software  expertise  is  necessary,  it  is  not  the  only  ingredient  in  the  recipe 
for  success.  Systems  engineering  is  also  a  critical  skill  and  an  inherent  PMO  responsibility. 
Therefore,  the  PMO  must  also  have  adequate  systems  engineering  staff.  In  addition,  for  soft¬ 
ware  critical  systems  expertise  is  needed  in  all  areas  including  software  management  (cost 
and  schedule  estimating,  metrics  definition  and  analysis,  process  improvement),  software 
acquisition,  software  architecture,  software  and  systems  integration,  software  development 
(including  commercial  off-the-shelf  [COTS],  reuse,  and  open  systems),  security  and  accredi¬ 
tation  and  organizational  change  management. 

The  expertise  needs  to  be  brought  on  early  in  the  program  to  develop  the  acquisition  strategy, 
support  RFP  preparations,  review  proposals,  contribute  to  program  development,  and  provide 
program  oversight.  The  staff  needs  to  maintain  their  skills  and  be  familiar  with  the  latest 
techniques  over  the  course  of  the  acquisition.  A  gap  analysis  should  be  performed  periodi¬ 
cally  to  identify  any  gaps  and  a  corresponding  training,  rotation  and  development  plans  put  in 
place  to  address  the  missing  skills. 

The  extent  to  which  the  PMO  has  the  appropriate  level  of  skills  it  needs  to  execute  the  acqui¬ 
sition  can  be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Weak  to 
Strong. 

3.4.1 .2  PMO  Staff  Capacity 

In  addition  to  having  the  appropriate  skills  as  described  in  Section  3. 4. 1.1,  the  PMO  also  has 
to  have  the  right  number  of  people  with  those  skills  to  plan  and  execute  the  acquisition.  It  is 
important  to  be  able  estimate  how  many  people  you  need,  with  what  skills,  and  at  what  time. 
For  example,  you  may  not  need  three  or  four  test  experts  while  you  are  preparing  the  acquisi¬ 
tion  strategy,  but  you  may  need  that  many  to  oversee  the  test  phase.  What  skills  does  the  pro¬ 
gram  need  when?  How  much  control  over  staff  capacity  and  the  escalation  process  for  resolv¬ 
ing  staffing  issues  that  are  out  of  your  control  does  the  program  have? 

The  degree  to  which  the  staff  members  have  worked  together  before  must  also  be  factored 
into  the  schedule.  Are  they  already  a  team  or  do  they  have  to  go  through  the  required  team¬ 
building  steps  before  they  function  optimally? 

The  extent  to  which  the  PMO  has  the  staff  capacity  it  needs  to  execute  the  acquisition  can  be 
rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Inadequate  to  Adequate. 

3.4.1. 3  PMO  Staff  Stability 

Permanent  Change  of  Station  (PCS)  and  Permanent  Change  of  Assignment  (PCA)  are  facts 
of  life  in  the  military.  The  program  must  take  this  into  account  when  formulating  its  acquisi¬ 
tion  strategy.  How  does  the  acquisition  strategy  compensate  for  the  fact  that  the  average  time 
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an  active  duty  person  spends  in  a  PMO  is  roughly  2-3  years?  Or,  that  the  higher  the  rank,  the 
more  likely  a  person  will  be  reassigned?  Programs  also  have  to  evaluate  the  stability  of  civil¬ 
ian  employees  or  contractors  supporting  the  PMO. 

How  will  continuity  and  history  be  maintained  for  the  program?  What  will  prevent  a  new 
software  architect  from  completely  changing  course?  What  will  help  a  new  integration  and 
test  lead  do  his  or  her  job  quickly  and  effectively  without  compromising  the  quality  of  the 
system?  Planning  for  continuity  in  the  event  of  personnel  reassignment  or  general  staff  reduc¬ 
tion  is  an  essential  part  of  acquisition  planning. 

The  extent  to  which  the  PMO  staff  is  stable  can  be  rated  on  a  sliding  scale  that  extends  from 
one  extreme  to  the  other:  Low  to  High. 

3.4.1. 4  PMO  Process  Focus 

A  process  focus  is  just  as  important  to  the  acquisition  organization  as  it  is  to  the  suppliers 
with  which  the  acquisition  organization  works.  Are  there  solid,  repeatable  processes  or  are 
things  done  in  an  ad  hoc  fashion?  Is  there  a  focus  on  improving  their  processes  through  the 
course  of  the  acquisition?  How  do  the  PMO  processes  mesh  with  the  supplier  processes?  The 
extent  to  which  the  PMO  understands  and  uses  process  and  process  improvement  initiatives 
during  the  acquisition  can  be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the 
other:  Weak  to  Strong. 

3.4.2  Stakeholders  Drivers 

Stakeholders  are  individuals  or  groups  with  vested  interest  in  the  system.  For  example,  end 
users,  system  integrators,  acquirers,  and  sponsors  are  all  stakeholders.  Each  of  these  stake¬ 
holders’  concerns  can  add  unique  risks,  many  of  which  are  discussed  in  other  sections.  For 
example,  how  willing  are  users  able  to  adapt  and  learn  a  new  system?  How  much  experience 
does  your  integrator  have  with  this  type  of  system?  How  well  staffed  is  the  acquisition  or¬ 
ganization?  How  well  do  the  sponsors  communicate  their  requirements? 

The  stakeholders  drivers  described  in  this  section  include  the  number  and  diversity  of  stake¬ 
holders,  the  level  of  stakeholder  engagement,  and  the  level  of  stakeholder  agreement. 

3.4.2. 1  Number  and  Diversity 

The  sheer  number  and  diversity  of  stakeholders  in  the  acquisition  impacts  the  acquisition 
strategy:  the  more  stakeholders  the  PMO  has  to  interface  with,  negotiate  with,  and  keep  in¬ 
formed,  the  higher  the  risk  for  the  acquisition.  It  is  essential  that  the  PMO  have  a  good  under¬ 
standing  of  how  many  stakeholders  are  involved  in  the  program.  A  complex  set  of  stake¬ 
holders  increases  software  risk  for  the  following  reasons. 

•  There  is  increased  probability  of  divergent  expectations  and  requirements  (see  Section 
3.5.1). 
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•  Tradeoffs  will  not  have  equal  impact  on  all  parties — there  will  be  winners  and  losers. 

•  Communication  becomes  more  complex 

•  Coordination  takes  more  time;  there  will  be  multiple  approval  paths  and  associated  bot¬ 
tlenecks. 

•  The  amount  of  education  required  to  discuss  software  issues  increases. 

Acquirers  must  be  careful  to  include  all  the  relevant  stakeholders,  since  omissions  can  affect 
drivers  and  risks.  This  does  not  imply,  however,  that  all  stakeholders  should  necessarily  have 
an  equal  voice  in  each  phase  of  the  acquisition. 

The  number  and  diversity  of  stakeholders  can  be  rated  on  a  sliding  scale  that  extends  from 
one  extreme  to  the  other:  Small  to  Large. 

3.4.2.2  Level  of  Engagement 

The  level  of  engagement  by  stakeholders  in  the  acquisition  process  can  affect  the  acquisition 
strategy.  One  way  to  describe  the  level  of  engagement  is  by  monitoring  the  scope  of  stake¬ 
holders’  involvement,  the  quality  of  their  input  and  feedback,  and  their  responsiveness.  In  the 
worst  case  scenario,  you  would  get  lots  of  input  in  an  untimely  fashion  that  is  of  poor  quality 
with  a  narrow  view  of  the  system.  In  the  best  case  scenario,  you  would  get  lots  of  input  in  a 
timely  fashion  that  is  of  high  quality  from  a  broad  view  of  the  system.  How  will  the  program 
ensure  proper  representation,  opportunities  for  participation,  and  commitment  to  deadlines 
from  stakeholders?  The  level  of  engagement  can  be  rated  on  a  sliding  scale  that  extends  from 
one  extreme  to  the  other:  Low  to  High. 

3.4.2. 3  Level  of  Agreement 

Stakeholder  agreement  is  the  amount  of  consensus  among  the  stakeholders  during  the  acqui¬ 
sition.  Stakeholders  should  be  proactively  included  in  all  phases  of  the  acquisition  to  maxi¬ 
mize  buy-in  and  system  acceptance.  The  impact  of  any  change  must  be  considered  from  all 
stakeholder  perspectives.  If  you  don’t  involve  them,  you  won’t  gain  commitment,  which  usu¬ 
ally  results  in  dissatisfaction  with  the  system  in  some  form,  may  cause  delays  in  the  program, 
and  so  on.  Even  if  stakeholders  are  proactively  included,  there  might  be  significant  disagree¬ 
ment  on  the  requirements  for  the  system,  the  priority  of  the  requirements,  on  the  schedule,  on 
the  deployment  plan,  and  so  on.  How  does  the  acquisition  strategy  handle  this? 

Another  aspect  to  consider  is  stakeholder  turnover.  Military  stakeholders  suffer  from  the 
same  PCS  and  PCA  factors  as  the  PMO.  Sometimes  a  change  in  a  key  stakeholder  can  force 
the  PMO  to  revalidate  decisions  that  were  made  earlier  in  the  program. 

As  agreement  decreases,  risk  increases  for  the  program.  The  level  of  agreement  can  be  rated 
on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Low  to  High. 
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3.4.3  Supplier  Capability  Drivers 

Acquisition  planning  does  not  stop  after  a  contract  award  is  made;  it  is  a  continuous  process. 
Many  programs  fail  to  evaluate  drivers  that  play  an  important  role  after  the  program  is  in 
execution.  A  category  of  drivers  that  is  often  not  considered  is  the  capability  of  the  supplier 
over  time. 

The  supplier  is  the  team  that  is  responsible  for  producing  the  system  and  its  associated  arti¬ 
facts.  Typically,  in  large  government  contracts  this  is  usually  one  or  more  contractors,  but 
that  is  not  always  the  case.  The  supplier  could  be  another  government  entity.  In  this  section, 
we  refer  to  suppliers  as  all  parties  responsible  for  supplying  the  system  to  the  acquirers. 

The  drivers  pertaining  to  supplier  capabilities  described  in  this  section  include  supplier  staff 
skills,  capacity  and  stability,  and  performance  to  date. 

3.4.3. 1  Supplier  Staff  Skills 

A  supplier  is  awarded  a  contract  based  on  its  ability  to  achieve  the  objectives  associated  with 
a  statement  of  work.  For  software-critical  systems,  expertise  may  be  needed  in  all  of  the  areas 
described  in  Section  3. 4. 1.1.  Other  expertise  may  also  be  needed,  depending  on  the  domain. 

During  program  execution,  an  evaluation  of  the  supplier  skill  set  is  critical  so  that  the  acquisi¬ 
tion  strategy  can  be  adjusted  to  mitigate  any  potential  risks  in  this  area.  For  example,  if  the 
supplier  lacks  testing  skills,  consider  hiring  an  independent  test  agency. 

The  extent  to  which  the  supplier  has  the  skills  needed  to  execute  its  responsibilities  on  the 
program  can  be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Weak  to 
Strong. 

3.4.3.2  Supplier  Staff  Capacity 

In  addition  to  having  the  right  skills,  the  supplier  also  has  to  have  the  right  number  of  people 
with  those  skills  at  the  right  time  to  meet  its  commitments  on  the  program.  The  extent  to 
which  the  supplier  has  the  staff  capacity  it  needs  to  meet  its  commitments  can  be  rated  on  a 
sliding  scale  that  extends  from  one  extreme  to  the  other:  Inadequate  to  Adequate. 

3.4.3. 3  Supplier  Staff  Stability 

Usually,  the  supplier  does  not  have  the  same  issues  with  PCSs  and  PCAs  as  the  PMO.  It 
does,  however,  have  to  manage  the  external  pressure  of  the  job  market  in  order  to  retain  key 
people.  Software  engineers,  system  engineers,  and  architects  are  usually  highly  sought  after 
resources.  The  hotter  the  job  market  in  certain  locations,  the  more  difficult  it  may  be  for  the 
supplier  to  retain  its  personnel. 
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Other  factors  that  cause  attrition  include  promotions  within  the  supplier  organization,  trans¬ 
ferring  to  the  latest  “hot”  program,  poor  program  management,  low  morale,  and  significant 
overtime. 

The  extent  to  which  the  supplier’s  staff  is  stable  can  be  rated  on  a  sliding  scale  that  extends 
from  one  extreme  to  the  other:  Low  to  High. 

3.4.3.4  Supplier  Performance  to  Date 

As  stated  in  Section  3.4.3. 1,  a  supplier  is  awarded  a  contract  based  on  its  ability  to  achieve 
the  objectives  associated  with  a  statement  of  work.  Typically,  some  form  of  past  performance 
criteria  are  used  during  the  evaluation  process.  During  contract  execution,  the  PMO  needs  to 
ensure  the  performance  of  the  supplier  does  not  add  additional  risk  to  the  program.  As  the 
program  progresses,  the  acquisition  strategy  can  be  modified  as  performance  weaknesses  or 
strengths  are  identified. 

The  extent  to  which  the  performance  of  the  supplier  to  date  can  be  rated  on  a  sliding  scale 
that  extends  from  one  extreme  to  the  other:  Poor  to  Excellent. 

3.5  Life  Cycle  Category 

Life  cycle-related  program  characteristics  and  associated  strategy  drivers  quantify  the  extent 
to  which  characteristics  of  various  life-cycle  phases  affect  the  software  risk  in  the  program. 
Life-cycle  strategy  drivers  include  product  definition  and  specification,  architecture  and  de¬ 
sign,  verification  and  test,  deployment,  operations  and  maintenance,  and  disposal. 

3.5.1  Product  Definition  and  Specification  Drivers 

A  great  deal  of  management  attention  is  placed  on  the  requirements  setting  phase 
because  missing,  vague  or  changing  requirements  tend  to  be  a  major  cause  of 
poor  software  development  outcomes  [GAO  04a], 

The  product  definition  and  specification  drivers  described  in  this  section  include  the  volatility 
and  understanding  of  the  requirements,  quality  attribute  definitions,  and  interoperability. 

3. 5. 1.1  Requirements  Volatility 

Stable,  fully  defined,  unambiguous,  consistent,  complete,  and  testable  software  requirements 
are  rare.  Having  flexible  requirements  is  not  a  bad  thing — trying  to  fully  define  software  re¬ 
quirements  too  early  or  trying  to  limit  requirements  changes  in  a  changing  environment  may 
be  riskier  than  allowing  some  level  of  flux.  The  acquisition  strategy  needs  to  accommodate 
the  degree  to  which  requirements  can  or  should  change  and  how  that  change  is  to  be  man¬ 
aged.  Lor  example,  the  acquisition  could  be  broken  into  several  phases  to  ensure  you  are 
ready  to  move  forward  and  are  ready  to  negotiate  or  compete  the  follow-on  phase. 
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Requirements  changes  can  come  from  multiple  sources  including  new  or  evolving  mission 
needs,  new  or  evolving  technology  (refresh  and  obsolescence),  modifications  to  budgets,  new 
or  evolving  policies  or  mandates,  and  changes  in  the  understanding  of  the  users  over  time. 

The  program  needs  to  understand  how  volatile  or  how  stable  the  requirements  for  the  system 
(and  software)  are  at  any  given  time  in  the  life  cycle.  The  amount  of  requirements  volatility 
has  a  significant  impact  on  the  acquisition  strategy  for  the  program.  Programs  seldom  have 
requirements  that  are  all  stable  or  all  unstable — some  requirements  are  firm  from  the  start, 
some  cannot  be  defined  until  other  things  about  the  system  are  known,  some  requirements 
may  be  in  a  constant  state  of  flux  as  technology,  COTS  products,  and  mission  needs  (or  the 
understanding  of  what  is  needed)  evolve.  There  may  be  problems  identifying  “all”  of  the 
stakeholders,  or  after  they’re  identified,  getting  them  involved  with  the  tradeoff  and  decision 
process.  What  one  stakeholder  wants  may  contradict  what  another  stakeholder  believes  to  be 
the  correct  product.  The  implication  is  that  the  acquisition  strategy  may  have  to  have  charac¬ 
teristics  of  either  extremes  or  compromise  between  them. 

How  volatile  are  the  system  requirements?  How  volatile  are  the  software  requirements?  How 
volatile  are  the  hardware  requirements?  If  hardware  requirements  are  stable  but  the  software 
requirements  are  not  stable  this  driver  can  be  split  into  hardware  requirements  volatility  and 
software  requirements  volatility  since  different  acquisition  strategies  could  potentially  be  ap¬ 
plied  to  the  hardware  and  software  components  of  the  acquisition.  The  same  can  be  said  for 
system-level  requirements  and  hardware/software  requirements.  Over  time,  the  volatility  of 
the  requirements  should  decrease  and  modifications  to  the  acquisition  strategy  can  be  made. 

If  the  volatility  of  the  requirements  does  not  decrease  as  expected,  this  is  a  good  indicator  that 
scope  is  not  well-defined  and  that  there  is  requirements  creep. 

The  extent  to  which  a  program’s  requirements  are  volatile  can  be  rated  on  a  sliding  scale  that 
extends  from  one  extreme  to  the  other:  Low  to  High. 

3. 5. 1.2  Requirements  Understanding 

As  stated  in  Section  3. 5. 1.1,  fully  defined,  unambiguous,  consistent,  complete,  and  testable 
software  requirements  are  rare.  Some  common  issues  with  the  understanding  of  software  re¬ 
quirements  include 

•  Requirements  are  specified  at  too  high  of  a  level  and  there  is  no  mechanism  to  refine 
them.  Vague  requirements  are  subject  to  multiple  interpretations. 

•  Ineffective  techniques  are  used  to  solicit  the  requirements  from  the  appropriate  stake¬ 
holders. 

•  The  operational  concept  is  not  sufficiently  detailed.  For  example,  the  operational  concept 
might  only  address  a  subset  of  the  users;  it  might  be  too  sparse,  which  leads  to  misinter¬ 
pretation,  or  it  might  contain  conflicting  requirements. 

•  Users  try  to  extend  the  functionality  of  the  system  past  the  “need  to  have”  state  into  the 
“want  to  have”  state. 

•  Interfaces  are  not  well  understood,  different  than  documented,  or  not  specified  at  all. 
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•  Stakeholders  are  not  involved  in  the  process  or  they  are  involved  and  the  importance  of 
their  input  is  ignored  or  minimized. 

•  Requirements  are  not  measurable,  with  clear  acceptance  criteria. 

•  A  common  vocabulary  is  not  shared  or  there  is  not  the  same  level  of  domain  expertise 
between  the  users,  acquisition  team,  and  performance  team/supplier.  For  example,  the  re¬ 
quirement  “Display  a  negative  number”  might  be  interpreted  by  a  developer  as  “use  a 
minus  sign”  when  in  reality,  the  user  is  an  accountant  and  is  expecting  parentheses  to  be 
used. 

The  extent  to  which  a  program’s  requirements  are  understood  by  the  users,  acquisition  or¬ 
ganization,  and  performance  team/supplier  can  be  rated  on  a  sliding  scale  that  extends  from 
one  extreme  to  the  other:  Low  to  Fligh. 

3.5. 1.3  Quality  Attribute  Definition  Quality 

Quality  attributes,  or  non-functional  requirements,  are  vitally  important  in  ensuring  appropri¬ 
ate  software  architecture.  Non-functional  requirements  are  used  to  express  some  of  the  “at¬ 
tributes  of  the  system”  or  “the  system  environment”  [Leffingwell  03].  They  are  usually  called 
the  “-ilities,”  quality  attributes,  or  general  requirements.  Examples  include  reliability,  seal- 
ability,  and  maintainability.  Not  all  quality  attributes  are  “-ilities”;  performance  and  security 
are  two  non-functional  requirements  or  quality  attributes.  For  example 

•  Flow  dependable  does  the  system  need  to  be? 

•  What  level  of  security  does  the  software  have  to  provide? 

•  What  availability  is  needed?  Flow  will  this  be  balanced  with  needed  maintenance  time 
frames?  Is  one  hour  a  month  downtime  for  maintenance  tolerable? 

Since  non-functional  requirements  are  requirements,  Sections  3. 5. 1.1  and  3. 5.2.2  apply  to  this 
type  of  requirement.  It  is  particularly  common  for  non- functional  requirements  to  be  defined 
without  measurable,  clear  acceptance  criteria.  For  example,  “the  page  loads  quickly”  is  in¬ 
adequate.  How  fast  must  the  page  load  under  what  conditions  on  the  system? 

Too  often,  requirements  are  thoroughly  defined  for  the  system  functionality  but  not  for  the 
quality  attributes.  Failure  to  define  the  attributes  the  system  needs  to  fulfill  puts  the  program 
at  great  risk.  How  do  you  know  which  quality  attributes  are  important  for  your  program? 

How  will  you  verify  that  your  program  has  addressed  its  quality  attribute  requirements  ade¬ 
quately?  The  extent  to  which  a  program’s  quality  attributes  are  defined  and  understood  can 
be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Low  to  High. 

3.5. 1.4  Interoperability 

Interoperability  is  the  ability  of  a  collection  of  communicating  entities  to  (a)  share  specified 
information  and  (b)  operate  on  that  information  according  to  an  agreed  operational  semantics. 
It  is  a  multi-dimensional  aspect  of  system  engineering.  The  scope  is  far  greater  than  simply 
interoperability  of  data.  It  includes,  among  other  things,  the  degree  of  coupling  and  owner- 
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ship.  It  also  includes  interoperability  at  the  organizational  and  management  levels  [Browns- 
word  04]. 

One  model  of  interoperability  examines  three  different  interoperability  dimensions: 

•  Programmatic  -  Activities  performed  to  manage  the  acquisition  of  a  system.  The  focus  is 
on  the  contracts,  incentives,  and  practices  such  as  risk  management.  It  is  very  important 
to  think  about  how  your  program  will  interface  to  other  programs.  Will  there  be  any  im¬ 
pact  on  your  schedule  due  to  delays  on  a  program  building  a  system/service  you  need? 

•  Constructive  -  Activities  performed  to  create  and  sustain  a  system.  The  focus  is  on  archi¬ 
tecture,  standards,  and  COTS.  How  many  interfaces  are  software-related  and  how  diffi¬ 
cult  are  they  to  implement?  How  will  you  govern  and  control  the  interfaces?  How  will 
testing  be  performed?  Does  the  information  mean  the  same  thing  in  my  system  as  in  the 
other  system(s)?  Which  is  the  best  system  service  to  use? 

•  Operational  -  Activities  performed  to  operate  a  system.  The  focus  is  on  interactions  with 
other  systems  (programs)  and  with  users.  How  will  the  concept  of  operations  be  man¬ 
aged?  How  will  consensus  be  reached  on  priorities?  What  is  the  appropriate  mix  of  capa¬ 
bilities  for  your  system?  [Morris  04] 

The  number  of  interfaces  and  their  characteristics  can  be  rated  on  a  sliding  scale  that  extends 
from  one  extreme  to  the  other:  Simple  to  Complex. 

3.5.2  Architecture  and  Design  Drivers 

The  software  architecture  plays  a  key  role  in  the  overall  system  design  because  it 

•  embodies  the  earliest  software  design  decisions  that  have  the  greatest  impact  on  the  sys¬ 
tem  and  may  be  difficult  to  specify  early  in  the  program 

•  largely  determines  a  system’s  ability  to  achieve  its  desired  qualities  (i.e.,  non-functional 
requirements)  such  as  performance,  security,  reliability,  modifiability,  and  others 

•  enables  or  inhibits  systematic  software  reuse  and  COTS  integration 

•  provides  a  means  to  analyze  and  predict  a  system’s  behavior 

•  enables  the  supplier  to  subdivide  work  appropriately 

The  software  architecture  must  be  developed  in  conjunction  with  the  systems  architecture  and 
software  issues  must  be  included  in  systems  engineering  tradeoffs. 

The  architecture  and  design  drivers  described  in  this  section  include  precedence,  quality  at¬ 
tribute  constraints,  technology  readiness,  legacy  system  considerations,  and 
COTS/government  off-the-shelf  (GOTS)/reuse  components.  These  drivers  can  be  affected  by 
other  drivers.  For  example,  policies  and  mandates  described  in  Section  3.2. 1  can  constrain  the 
architecture. 
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3.5.2. 1  Precedence 


The  degree  to  which  the  system  is  precedented  or  unprecedented  has  a  significant  impact  on 
the  risk  of  a  program.  According  to  the  Encyclopedia  of  Software  Engineering,  precedented 
systems  are  those  for  which 

•  the  requirements  are  consistent  and  well-understood 

•  the  system  architecture  (both  hardware  and  software)  is  known  to  be  adequate  for  the 
requirements 

•  the  acquisition  and  development  teams  have  worked  together  to  build  a  previous  similar 
system  [Marciniak  05] 

Violation  of  one  or  more  of  these  causes  the  system  to  be  classified  as  unprecedented. 

Clearly,  we  are  better  at  producing  some  types  of  software  than  others.  We  can  consistently 
produce  small  or  common  software  applications.  In  fact,  a  commercial  marketplace  has  de¬ 
veloped  to  provide  these  software  components.  On  the  other  hand,  we  are  less  capable  of 
producing  large,  unprecedented  systems  where  few  ready-made  components  are  available  or 
where  the  complexity  of  creating  a  unified  system  from  components  is  stretched  past  previ¬ 
ous  limits. 

Major  stumbling  blocks  for  unprecedented  software-critical  systems  include 

•  ill-defined,  incomplete,  contradictory,  and  changing  requirements 

•  inadequate  architecture 

•  software  and  hardware  architecture  mismatches 

•  lack  of  domain  experience 

Relatively  simple  systems  present  relatively  little  risk.  Similar  systems  can  be  used  as 
benchmarks  for  reasoning  about  the  system  underdevelopment.  Unprecedented  systems  obvi¬ 
ously  present  much  greater  risk  in  that  there  are  no  equivalent  benchmarks  from  which  to 
draw  inferences.  A  program  can  only  extrapolate  from  other  related  systems.  The  more  un¬ 
precedented  a  system,  the  more  risk  for  the  program.  The  degree  to  which  a  system  is  prece¬ 
dented  can  be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Low  to 
High. 

3. 5.2.2  Quality  Attribute  Constraints 

The  definition  of  quality  attributes  was  discussed  in  Section  3.5. 1.3.  The  system  can  have 
simplistic  qualities  and  few  consequences  for  failure,  very  demanding  qualities  and  severe 
consequences  for  failure,  or  be  somewhere  in  between.  The  quality  attributes  can  put  signifi¬ 
cant  constraints  on  how  the  system  can  be  architected  and  implemented.  The  extent  to  which 
a  program’s  quality  attribute  requirements  constrain  the  solution  set  can  be  rated  on  a  sliding 
scale  that  extends  from  one  extreme  to  the  other:  Low  to  High. 
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3. 5.2. 3  Technology  Readiness 

Developing  software  for  a  system  depends  on  many  technologies.  Is  the  hardware  mature 
enough  for  the  software  to  be  developed?  Is  the  development  environment  including  compil¬ 
ers,  configuration  management  tools,  and  other  development  tools  sufficient  to  support  the 
software  development  effort?  Developing  software  also  depends  on  the  specific  operational 
context.  For  example,  Java  is  fairly  mature  at  this  point,  but  there  are  still  some  applications 
where  it  is  inappropriate  to  use. 

The  extent  to  which  system  technologies  are  mature  enough  for  software  to  be  developed  can 
be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Immature  to  Mature. 

3.5.2.4  Legacy  Considerations 

Legacy  system  considerations  introduce  many  avenues  for  risk  during  an  acquisition  includ¬ 
ing  how  well  the  legacy  system  and  its  interfaces  are  documented  and  managed,  the  amount 
of  change  between  a  legacy  system  and  its  replacement  system,  backward  compatibility  re¬ 
quirements,  and  so  on.  These  are  just  a  few  of  the  many  considerations,  but  they  highlight  the 
risks  that  programs  that  have  legacy  requirements  must  address. 

How  well  the  legacy  system  is  understood  significantly  influences  risk.  If  a  legacy  system 
and  its  interfaces  are  poorly  documented  there  could  be  many  interoperability  and  replace¬ 
ment  issues.  Is  the  legacy  system  written  in  some  outdated  or  obscure  programming  lan¬ 
guage?  How  accurate  is  the  information  provided?  How  hard  will  it  be  and  how  long  will  it 
take  to  obtain  any  additional  information  your  program  requires?  Does  the  government  have 
full  data  rights  to  the  existing  system?  How  long  will  you  have  to  interoperate  with  a  legacy 
system?  How  will  you  keep  up  with  the  changes  to  the  legacy  system? 

There  are  also  transition  risks  as  you  move  from  a  working  legacy  system  to  a  new  system. 
Do  the  two  systems  need  to  interoperate  for  some  time  or  does  the  new  system  completely 
replace  the  old  system?  Will  there  be  a  new  user  interface  to  learn?  Are  there  any  safety  or 
security  risks  as  a  result?  Are  there  any  facilities  issues  caused  by  the  transition  needs? 

The  requirement  to  be  backward  compatible  with  a  prior  or  legacy  system  adds  its  own  set  of 
risks,  especially  when  the  earlier  system  still  exists  and  is  undergoing  changes.  How  will  new 
requirements  in  the  legacy  system  be  coordinated  and  managed  with  the  new  one?  Is  an  ap¬ 
propriate  governance  board  in  place?  How  different  can  the  new  system  be  from  the  prior 
one?  How  long  will  backward  compatibility  need  to  be  maintained  after  deployment? 

The  complexity  of  legacy  system  aspects  that  must  be  considered  can  be  rated  on  a  sliding 
scale  that  extends  from  one  extreme  to  the  other:  Low  to  High. 
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3. 5.2. 5  COTS,  GOTS,  and  Software  Reuse 

Maximizing  COTS/GOTS/reuse  is  often  a  goal  in  programs  today.  The  implications  of  using 
COTS/GOTS/reuse  must  be  fully  considered.  An  acquisition  program  must  have  their  “eyes 
wide  open”  to  understand  the  benefits  and  mitigate  the  risks. 

COTS  or  GOTS  product  use  within  the  system  may  be  considered  a  low-risk  alternative  for 
implementing  specific  software  driven  portions  of  the  product  design.  However, 

COTS/GOTS  should  be  considered  as  risky  as  any  other  software.  The  off-the-shelf  options 
should  not  be  considered  “low  risk”  simply  because  the  product  can  be  purchased  or  obtained 
as  a  pre-packaged  unit.  COTS/GOTS  products  are  not  always  complete  or  completely  tested 
and  most  often  are  not  based  on  the  requirements  for  the  new  system  being  built.  Planning  for 
COTS/GOTS  use,  including  a  maintenance  strategy,  is  an  important  part  of  up-front  program 
planning. 

The  use  of  COTS/GOTS  can  be  beneficial  but  it  does  bring  some  challenges  with  it.  Some 
key  challenges  include  [Carney  03] 

•  COTS  products  are  driven  by  the  marketplace,  not  the  system  context 

•  built-in  assumptions  of  end-user  processes  that  may  not  match  yours 

•  licensing,  data  rights,  and  warranties 

•  limited  control  of  content  or  frequency  of  releases 

•  limited  visibility  into  product  internals  and  behavior 

•  varying  architectural  paradigms  between  the  product  and  other  system  components 

•  GOTS  sustainment  responsibilities  and  plans 

The  way  reuse  is  defined,  planned  for  (e.g.,  productivity  savings,  maintenance),  and  managed 
determines  the  risk  involved  with  reuse.  Effective  reuse  typically  involves  more  than  just 
source  code;  design,  requirements,  test  scripts  can  all  be  reused  as  well.  The  degree  to  which 
the  characteristics  of  the  program’s  intended  operational  context  are  similar  to  the  operational 
context  for  which  the  original  code  was  intended  is  one  determinant  of  the  risk  involved  with 
reuse. 

Code  reuse  can  come  in  many  forms  including  reuse  of  legacy  code  and  reuse  of  code  be¬ 
tween  components/modules  of  the  system  being  acquired.  One  must  also  take  into  account 
that  such  reuse  is  rarely  ever  “pure”  (meaning  that  the  code  is  reused  as-is  without  any  modi¬ 
fication).  More  often  the  reused  code  is  modified  in  some  way  to  meet  the  specific  require¬ 
ments  of  the  system  under  development  or  the  reused  must  be  “wrapped”  by  newly  devel¬ 
oped  code  to  provide  a  consistent  interface  to  the  rest  of  the  system.  In  either  case,  there  is 
new  development  involved  with  the  reuse  and  the  amount  of  that  new  development  will  off¬ 
set  (at  least  partially)  the  benefits  of  the  reuse. 
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The  extent  to  which  COTS/GOTS/reuse  is  planned  to  provide  system  functionality  and  real¬ 
ized  over  time  can  be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Low 
to  High. 

3.5.3  Verification  and  Test  Drivers 

The  verification  and  test  drivers  described  in  this  section  include  the  complexity  and  avail¬ 
ability  of  the  test  environment  and  the  number  of  system  configurations. 

3.5.3. 1  Test  Environment  Complexity 

Typically,  as  the  test  environment  complexity  increases,  the  risk  also  increases.  Do  you  need 
a  special  environment  to  perform  software  testing  such  as  an  aircraft?  Is  a  classified  test  envi¬ 
ronment  required?  Is  destructive  testing  required?  Is  the  software  part  of  a  self-destruct  capa¬ 
bility  of  an  end  item?  How  complex  are  the  test  tools?  Has  appropriate  training  been  pro¬ 
vided?  Is  the  test  environment  spread  over  several  geographical  locations  that  must  be 
connected  via  a  network?  Are  some  of  the  elements  of  the  test  environment  owned  by  other 
entities,  for  example,  a  government  laboratory?  Will  simulations  (of  nascent  hardware  or  of 
battlefield  conditions)  need  to  be  developed? 

The  complexity  of  the  test  environment  can  be  rated  on  a  sliding  scale  that  extends  from  one 
extreme  to  the  other:  Low  to  High. 

3. 5. 3.2  Test  Environment  Availability 

Typically,  as  the  availability  of  the  necessary  test  environment  decreases,  the  risk  increases. 
Can  you  get  the  testing  time  required  in  the  necessary  test  environment?  What  are  the  priori¬ 
ties  and  schedule  constraints  on  the  test  agency  and  facilities?  How  long  does  it  take  to  get 
into  the  test  cycle?  Does  this  match  up  with  your  schedule?  Is  another  program  ahead  of  you 
using  the  test  environment?  What  is  the  likelihood  of  slippage  into  your  schedule  time? 

In  addition  to  the  agency  and  facilities  for  testing,  you  also  have  to  consider  the  components 
with  which  you  need  to  test.  For  example,  if  you  performed  some  testing  that  interfaces  with 
an  external  element  that  is  a  prototype  or  engineering  model,  what  is  the  risk  that  the  final 
version  of  the  element  will  differ  from  what  you  tested  with? 

The  availability  of  the  necessary  test  environment  can  be  rated  on  a  sliding  scale  that  extends 
from  one  extreme  to  the  other:  Low  to  High. 

3. 5. 3. 3  Number  of  System  Configurations 

The  number  of  system  configurations  has  an  impact  on  the  test  requirements  and  how  they 
will  be  tested.  Tradeoffs  will  need  to  be  made  regarding  the  test  requirements  of  the  different 
configurations  given  the  amount  of  funding  available  for  testing.  Can  the  same  test  cases  be 
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used  or  do  different  test  cases  need  to  be  developed?  What  about  test  simulators?  Can  they  be 
parameterized  or  do  they  need  to  be  different? 

Even  if  you  only  have  one  configuration  under  test,  ensuring  that  the  configuration  is  not 
modified  during  the  testing  process  is  not  always  easy.  For  example,  a  testbed  was  used  dur¬ 
ing  the  day  for  testing  the  system  configuration  on  a  project.  In  the  evening  it  was  used  for 
training  purposes.  Between  the  daily  testing  cycles,  the  test  system  configuration  was 
changed  by  the  training  team  without  the  test  team’s  knowledge.  How  valid  were  the  tests? 
The  importance  of  managing  the  configuration  of  the  system,  test  enviromnent,  and  other  test 
artifacts  is  essential. 

The  number  of  system  configurations  that  need  to  be  tested  can  be  rated  on  a  sliding  scale 
that  extends  from  one  extreme  to  the  other:  Few  to  Many. 

3.5.4  Deployment  Drivers 

The  deployment  drivers  described  in  this  section  include  the  number  of  sites  and  system  con¬ 
figurations,  user  and  maintainer  readiness,  and  data  migration  considerations. 

3. 5.4.1  Number  of  Sites 

Sites  may  have  different  environments  (hardware,  software,  networking,  etc.)  that  impact  the 
deployment.  Personnel  may  have  different  levels  of  expertise  at  different  sites.  The  approval 
process  or  acceptance  process  may  be  different.  Another  factor  to  consider  is  the  timing  for 
the  deployments  if  there  are  multiple  sites.  If  there  is  schedule  overlap,  staffing  considera¬ 
tions  must  be  taken  into  account. 

The  number  of  sites  the  system  will  be  deployed  to  can  be  rated  on  a  sliding  scale  that  ex¬ 
tends  from  one  extreme  to  the  other:  Few  to  Many. 

3. 5.4.2  User  Readiness 

How  ready  the  users  are  to  accept  and  use  the  system  is  critical  to  the  success  of  the  system  in 
fulfilling  the  mission.  Some  software  considerations  include 

•  Have  all  operational  software  training  needs  been  identified? 

•  Will  any  new  training  facilities  or  simulators  be  needed? 

•  Does  the  life  cycle  support  the  development  of  software  for  training  assets? 

•  Are  the  organizations  impacted  involved  throughout  the  life  cycle? 

•  Will  there  be  changes  in  personnel  allocations  to  operate/maintain  the  system? 

The  extent  to  which  users  have  the  skills  and  desire  to  use  the  system  can  be  rated  on  a  slid¬ 
ing  scale  that  extends  from  one  extreme  to  the  other:  Low  to  High. 
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3. 5.4.3  Maintainer  Readiness 


How  ready  the  maintainers  are  to  operate  and  maintain  the  system  is  also  critical  to  the  mis¬ 
sion.  Some  software  considerations  include 

•  Have  all  maintenance  software  training  needs  identified? 

•  Will  any  new  tools  be  needed?  Does  the  life  cycle  support  the  development  of  software 
for  the  new  tools? 

•  Does  the  life  cycle  support  the  development  of  software  for  training  assets? 

•  Are  the  organizations  impacted  involved  throughout  the  life  cycle? 

The  extent  to  which  maintainers  have  the  skills,  tools,  and  desire  to  operate  and  maintain  the 
system  can  be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Low  to 
High. 

3. 5.4.4  Transition  and  Data  Migration 

Data  migration,  data  base  schema  and  data  dictionary  changes,  access  modifications,  and 
other  technical  transition  issues  can  increase  risk  in  an  acquisition  program.  As  the  number  of 
new  or  changing  technical  elements  increases,  risk  can  compound.  How  will  the  transition  be 
staged?  Will  parallel  operations  between  the  old  applications  or  databases  and  the  new  ones 
be  needed?  Will  data  be  migrated  manually  or  will  it  be  automated?  How  will  the  integrity  of 
the  data  and  the  system  operations  be  verified?  Is  the  support  infrastructure  in  place  to  sup¬ 
port  the  change?  Is  there  a  contingency  plan? 

The  extent  that  technical  transition  issues  need  to  be  addressed  can  be  rated  on  a  sliding  scale 
that  extends  from  one  extreme  to  the  other:  Low  to  High. 

3.5.5  Maintenance  and  Support  Drivers 

The  maintenance  and  support  drivers  described  in  detail  in  this  section  include  the  number  of 
system  configurations,  update  readiness,  support  duration,  re-competition  readiness,  opera¬ 
tional  environment,  legacy  system  support  considerations  and  availability  of  data  rights. 

3.5.5. 1  Number  of  System  Configurations 

Does  the  system  require  different  software  for  different  sites?  If  so,  how  many  configurations 
need  to  be  supported?  The  number  of  system  configuration  that  will  be  deployed  can  be  rated 
on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Few  to  Many. 

3. 5. 5.2  Update  Readiness 

System  updates  are  inevitable.  How  frequently  the  system  is  updated,  when,  how,  and  by 
who  are  all  factors  that  must  be  considered.  Usually,  as  the  number  of  updates  increases,  so 
does  the  risk.  But  there  are  other  risk  factors  to  consider  when  updating  software.  For  exam- 
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pie,  lack  of  an  adequate  configuration  control  process  for  software  changes  could  cause 
havoc  even  if  only  one  software  update  was  applied.  Other  factors  include 

•  Is  software  maintenance  adequately  planned  and  emphasized? 

•  Is  the  software  maintainable? 

•  Can  the  system  and  software  architecture  support  future  changes? 

•  Are  support  assets  such  as  training  and  documentation  being  updated  concurrently? 

•  Are  the  software  development  environment  and  test  tools  available  to  the  maintainers? 

The  extent  to  which  the  program  is  ready  to  update  the  system  can  be  rated  on  a  sliding  scale 
that  extends  from  one  extreme  to  the  other:  Low  to  High. 

3. 5. 5. 3  Support  Duration 

The  length  of  time  the  system  will  be  in  sustainment  must  be  considered  when  developing 
and  modifying  your  acquisition  strategy.  What  effort  is  required  to  avoid  decay  and  obsoles¬ 
cence?  How  will  obsolescence  of  COTS  products  or  other  enabling  technologies  be  planned 
and  managed?  How  will  you  ensure  support  continuity?  Is  it  better  to  use  your  development 
contractor  or  a  depot?  If  the  system  was  initially  developed  with  a  5 -year  window  and  now 
you  are  asked  to  sustain  it  for  1 0  years  are  you  going  to  change  anything?  Do  you  have  a  sys¬ 
tem  maintenance  plan? 

The  length  of  time  the  system  is  planned  to  be  in  sustainment  can  be  rated  on  a  sliding  scale 
that  extends  from  one  extreme  to  the  other:  Short  to  Long. 

3. 5. 5.4  Re-Competition  Readiness 

If  your  acquisition  strategy  includes  successive  competitions  or  you  are  forced  into  a  com¬ 
petitive  situation  (i.e.,  due  to  performance)  will  you  be  ready  to  re-compete  the  contract  when 
the  time  comes?  Is  system  documentation  up-to-date?  Are  the  technologies  supportable?  For 
example,  when  the  Y2K  crisis  arose,  there  was  great  demand  for  support  options  that  were  in 
limited  supply.  How  will  support  options  and  their  availability  change  during  contingency 
operations? 

The  extent  to  which  you  are  ready  to  re-compete  your  acquisition  in  whole  or  part  can  be 
rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Low  to  High. 

3. 5. 5. 5  Operational  Environment 

The  environment  in  which  the  system  will  operate  can  have  an  impact  on  your  acquisition 
strategy.  The  operational  environment  can  include  the  natural  environment  (e.g.,  space  or 
desert),  user  conditions  (e.g.,  immigrant  workers  with  well  worn  fingerprints  crossing 
through  a  border  checkpoint  that  uses  fingerprints  as  an  identification  mechanism),  and  re¬ 
quirements  of  the  computer  or  other  devices  the  software  interacts  with. 
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You  must  understand  the  conditions  or  constraints  under  which  the  system  will  operate.  For 
example,  does  the  software  test  for  boundary  conditions  related  to  the  environment? 

The  constraints  of  the  operational  environment  can  be  rated  on  a  sliding  scale  that  extends 
from  one  extreme  to  the  other:  Benign  to  Flarsh. 

3. 5. 5. 6  Legacy  Considerations 

Legacy  system  considerations  were  discussed  in  Section  3. 5.2. 4.  During  the  maintenance  and 
support  part  of  the  life  cycle,  additional  legacy  system  considerations  must  be  taken  into  ac¬ 
count.  What  happens  if  a  legacy  system  your  system  depends  on  will  no  longer  be  supported? 
Is  the  maintenance  schedule  dependent  on  a  legacy  system  update  or  vice  versa?  Are  there 
interface  changes  that  need  to  be  supported? 

The  complexity  of  the  legacy  system  aspects  that  must  be  considered  during  maintenance  and 
support  can  be  rated  on  a  sliding  scale  that  extends  from  one  extreme  to  the  other:  Low  to 
High. 

3. 5. 5. 7  Complexity  of  Data  Rights 

The  availability  of  data  rights  is  a  complex  aspect  of  acquisition,  particularly  as  systems  con¬ 
tinue  to  grow  in  size  and  their  use  of  COTS  software,  GOTS  software,  free  and  open  source 
software,  and  more.  The  Council  on  Governmental  Relations2  defines  data  rights  as  the  “li¬ 
cense  the  government  obtains  in  technical  data  which  grantees  and  contractors  deliver  to  the 
government  after  completion  of  the  project.”  It  goes  on  to  state  that  what  makes  it  so  com¬ 
plex  are  the  gradations  in  scope  from  non-exclusive  to  exclusive  license  rights.  There  are  also 
differences  between  how  DoD  and  civilian  agencies  determine  allocation  of  rights  and  the 
definition  of  technical  data  rights  versus  the  term  “computer  software.”  In  addition,  FAR 
Subpart  27.4  is  devoted  to  data  rights. 

The  complexity  of  data  rights  issues  that  must  be  considered  can  be  rated  on  a  sliding  scale 
that  extends  from  one  extreme  to  the  other:  Low  to  High. 

3.5.6  Disposal  Driver 

It  might  not  be  readily  apparent  that  software  has  an  impact  on  disposal,  but  there  are  some 
disposal  issues  with  software.  Classification  is  probably  the  largest  concern.  Are  there  any 
classification  concerns  with  the  software  in  the  system?  Another  issue  is  the  ability  to  remove 
software  from  the  system.  How  will  it  be  removed?  Will  its  removal  affect  other  systems  op¬ 
erating  on  the  same  hardware? 

The  actions  taken  during  the  acquisition  can  make  the  activities  to  ensure  the  disposal  of  de¬ 
commissioned,  destroyed,  or  irreparable  system  components  meet  all  applicable  regulations 


2  For  more  information,  visit  http://www.cogr.edu/docs/RightsComputer.htm. 
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less  or  more  difficult.  As  the  acquisition  strategy  evolves  over  time,  software  implications  for 
disposal  should  not  be  ignored. 

The  extent  to  which  a  program  has  software  disposal  considerations  can  be  rated  on  a  sliding 
scale  that  extends  from:  Unrestricted  to  Restricted. 
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4  Key  Acquisition  Strategy  Elements 


4.1  Strategy  Element  Overview 

All  strategies  can  be  plans,  but  not  all  plans  can  be  strategies.  In  the  context  of  acquisition, 
strategies  are  high-level  decisions  that  direct  the  development  of  more  detailed  plans  that 
guide  the  execution  of  a  program.  Careful  planning  of  what  is  to  be  done,  who  will  do  it,  and 
when  it  will  be  done  is  required. 

Developing  an  all-encompassing  acquisition  strategy  for  a  program  is  a  daunting  activity.  As 
with  many  complex  endeavors,  often  the  best  way  to  begin  is  to  break  the  complex  activity 
down  into  simpler,  more  manageable  tasks.  Our  approach  to  developing  an  acquisition  strat¬ 
egy  is  not  to  craft  a  monolithic  plan,  but  to  a  carefully  integrate  a  collection  of  individual 
strategy  elements  and  corresponding  strategic  choices  tailored  to  address  the  program  drivers. 
Thus,  when  developing  an  acquisition  strategy,  a  program  manager’s  first  task  is  to  define  the 
elements  of  that  strategy.  When  defining  strategy  elements,  it  is  useful  to  ask  the  question, 
“What  acquisition  choices  must  I  make  in  structuring  this  program?”  Inevitably,  asking  this 
question  leads  to  more  detailed  questions  such  as 

•  What  acquisition  approach  should  I  use? 

•  What  type  of  solicitation  will  work  best? 

•  How  will  I  monitor  my  contractor’s  activities? 

•  and  many  more 

The  result  of  these  choices  defines  the  acquisition  strategy.  Identifying  the  strategy  elements 
is  the  first  step  in  this  process.  Several  resources  exist  to  aid  us  in  this  task. 

The  Defense  Acquisition  University’s  (DAU)  Defense  Acquisition  Guidebook  defines  19 
“strategy  considerations”  that  should  be  addressed  when  formulating  an  acquisition  strategy 
[DAU  04].  Many  of  these  considerations,  listed  in  Table  3,  are  composed  of  lower-level  sup¬ 
porting  considerations.  Each  consideration  requires  careful  attention  by  the  program  man¬ 
agement  team  and  should  be  addressed  within  the  acquisition  strategy. 
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Table  3:  DAU  Defense  Acquisition  Guidebook  Acquisition  Strategy  Considerations 


Acquisition  Strategy 
Consideration 

Supporting  Considerations 

Program  Structure 

•  Milestone  Decision  Points  •  Acquisition  Phases 

Acquisition  Approach 

Capability  Needs 

Test  and  Evaluation 

Risk  Management 

•  Risk  Management  in  Systems  •  Risk  Management  in  Program 

Engineering  Protection 

Resource  Management 

•  Cost  Estimation  •  Advance  Procurement 

•  Funding 

Systems  Engineering  Plan 

•  Technical  Baseline  Management  •  Metrics 

•  Technical  Reviews 

Interoperability 

•  Information  Interoperability  •  Other  Interoperability  (intersystem, 

interprogram) 

Information  Technology 

•  Infrastructure  Considerations  •  Support  Considerations 

Research  and  Technology 
Protection 

•  Protection  of  Critical  Program  •  Anti-Tamper  Measures 

Information 

Information  Assurance 

Product  Support  Strategy 

•  Product  Support  (including  software)  •  Environment,  Safety,  and  Occupational 

•  Interoperability  Health  (ESOH) 

•  Data  Management  (DM)  •  Human  Systems  Integration  (HSI) 

•  Integrated  Supply  Chain  Management  •  Maintain  Designated  Science  and 

•  Life  Cycle  Cost  Optimization  Technology  Information,  the  Security 

•  Logistics  Footprint  Minimization  Classification  Guide,  and  the  Counter- 

•  Life  Cycle  Assessment  intelligence  Support  Plan. 

•  Demilitarization  and  Disposal 

Human  Systems  Integration 

•  Manpower  •  Safety  and  Occupational  Health 

•  Personnel  •  Personnel  Survivability 

•  Training  •  Habitability 

•  Human  Factors 

Environment,  Safety,  and 
Occupational  Health  (ESOH) 

Modular  Open  Systems 
Approach  (MOSA) 

•  MOSA  Integration  •  MOSA  Monitoring 

•  MOSA  Development 

Business  Considerations 

•  Competition  •  Leasing 

•  International  Cooperation  •  Equipment  Valuation 

•  Contract  Approach 

Best  Practices 

•  Integrated  Product  and  Process  •  Realistic  Cost  Estimates  and  Cost 

Development  Objectives 

•  Performance-Based  Specifications  •  Adequate  Competition  Among  Viable 

•  Management  Goals  Offerors 

•  Reporting  and  Incentives  •  Best  Value  Evaluation  and  Award 

•  Modular  Open  Systems  Approach  Criteria 

•  Replacement  of  Government-Unique  •  Use  of  Past  Performance  in  Source 

Management  and  Selection 

Manufacturing  Systems  •  Software  Capability  Evaluations 

•  Technology  Insertion  for  Continuous  •  Government-Industry  Partnerships 

Affordability  Improvement 

Relief,  Exemption,  or  Waiver 

Additional  Acquisition 

Strategy  Topics 

•  Program  Office  Staffing  and  Support  •  Simulation  Based  Acquisition  and 

Contractor  Resources  Available  to  the  Modeling  and  Simulation 

Program  Manager  •  Software-Intensive  Programs  Review 

•  Integrated  Digital  Environment 

Management 

•  Government  Property  in  the  Possession 
of  Contractors  Management 
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Another  resource  is  the  CMMI  Acquisition  Module  (CMMI-AM),  Version  1.1  [Bernard  05]. 
As  shown  in  Table  4,  this  interpretive  guide  to  the  use  of  the  CMMI  framework  in  an  acquisi¬ 
tion  environment  defines  12  Process  Areas  to  be  addressed  by  the  acquirer.  Each  of  these 
process  areas  should  also  be  addressed  by  the  acquisition  strategy. 
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Table  4:  CMMI-AM  and  Acquisition  Strategy  Elements 


CMMI-AM 
Process  Area 

Goals 

Project 

Planning 

•  Estimates  of  project  planning  pa¬ 
rameters  are  established  and  main¬ 
tained. 

•  A  project  plan  is  established  and 
maintained  as  the  basis  for  manag¬ 
ing  the  project. 

•  Commitments  to  the  project  plan  are 
established  and  maintained. 

Project 
Monitoring 
and  Control 

•  Actual  performance  and  progress  of 
the  project  are  monitored  against  the 
project  plan. 

•  Corrective  actions  are  managed  to 
closure  when  the  project’s 
performance  or  results  deviate 
significantly  from  the  plan. 

•  Work  is  coordinated  with  suppliers 
to  ensure  the  contract  is  executed 
properly. 

Solicitation 
and  Contract 
Monitoring 

•  The  project  is  prepared  to  conduct 
the  solicitation. 

•  Suppliers  are  selected  based  on  the 
solicitation  package. 

•  Contracts  are  issued  based  on  the 
needs  of  the  acquisition  and  the 
suppliers’  proposed  approaches. 

Integrated 

Project 

Management 

•  The  project  is  conducted  using  a 
defined  process  that  is  tailored  from 
the  organization’s  set  of  standard 
processes. 

•  Coordination  and  collaboration  of 
the  project  with  relevant  stake¬ 
holders  are  conducted. 

Risk 

Management 

•  Preparation  for  risk  management  is 
conducted. 

•  Risks  are  identified  and  analyzed  to 
determine  their  relative  importance. 

•  Risks  are  handled  and  mitigated, 
where  appropriate,  to  reduce  adverse 
impacts  on  achieving  objectives. 

Requirements 

Development 

•  Stakeholder  needs,  expectations, 
constraints,  and  interfaces  are 
collected  and  translated  into 
customer  requirements. 

•  Customer  requirements  are  refined 
and  elaborated  to  develop  product 
and  product/component 
requirements. 

•  The  requirements  are  analyzed  and 
validated,  and  a  definition  of  re¬ 
quired  functionality  is  developed. 

CMMI-AM 
Process  Area 

Goals 

Requirements 

Management 

•  Requirements  are  managed  and 
inconsistencies  with  project 
plans  and  work  products  are 
identified. 

Verification 

•  Preparation  for  verification  is 
conducted. 

•  Peer  reviews  are  performed  on 
selected  work  products. 

•  Selected  work  products  are 
verified  against  their  specified 
requirements. 

Validation 

•  Preparation  for  validation  is 
conducted. 

•  The  product  or  product  compo¬ 
nents  are  validated  to  ensure  that 
they  are  suitable  for  use  in  their 
intended  operating  environment. 

Decision 
Analysis  and 
Resolution 

•  Decisions  are  based  on  an 
evaluation  of  alternatives  using 
established  criteria. 

Measurement 
and  Analysis 

•  Measurement  objectives  and 
activities  are  aligned  with 
identified  information  needs  and 
objectives. 

•  Measurement  results  that  ad¬ 
dress  identified  information 
needs  and  objectives  are 
provided. 

Transition  to 
Operations 
and  Support 

•  Preparation  for  transition  to 
operations  and  support  is 
conducted. 

•  Transition  decisions  and  actions 
are  executed  in  accordance  with 
transition  criteria. 
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By  examining  the  strategy  considerations  shown  in  Table  3,  we  find  a  mixture  of  both  strat¬ 
egy  drivers  and  elements  of  an  acquisition  strategy.  For  example,  the  Acquisition  Approach 
is  a  decision  that  must  be  made  by  the  program  manager — it  is  a  strategy  element.  Similarly, 
the  Product  Support  Strategy  is  a  collection  of  decisions  to  be  made — it  is  a  strategy  element. 
Other  considerations,  such  as  Interoperability  and  Capability  Needs  are  not  necessarily  deci¬ 
sions  to  be  made  by  the  program  manager,  but  are  factors  that  influence  the  execution  of  the 
project;  therefore,  they  are  program  drivers. 

Likewise,  the  CMMI-AM  outlined  in  Table  4  does  not  clearly  identify  acquisition  strategy 
elements.  It  lists  the  goals  to  be  accomplished,  but  says  nothing  about  the  strategies  employed 
to  accomplish  them. 

While  both  the  Defense  Acquisition  Guidebook  and  the  CMMI-AM  are  helpful  in  framing  the 
concepts  behind  developing  an  acquisition  strategy,  neither  explicitly  defines  nor  differenti¬ 
ates  the  drivers  and  the  elements  of  an  acquisition  strategy  in  a  manner  suitable  for  using  a 
specific  method  to  develop  a  strategy.  However,  using  both  of  these  resources  as  a  reference, 
we  have  created  a  partial  list  of  strategy  elements  that  could  be  applied  methodically  to  strat¬ 
egy  development.  These  elements  are  shown  in  Table  5;  boldfaced  text  denotes  the  elements 
and  sub-elements  discussed  in  this  report. 


Table  5:  Partial  List  of  Strategy  Elements  and  Sub-Elements 


Strategy  Element 

Strategy  Sub-Element 

Program  Structure 

Milestone  Decision  Points 

Acquisition  Phases 

Acquisition  Approach 

Business  Considerations 

Competition 

Solicitation 

Source  Selection 

Contract  Approach 

Risk  Management 

Information  Assurance 

Supplier  Assurance 

Test  and  Evaluation 

Product  Support 

Training 

Installation 

Source  of  Support 

This  remainder  of  this  section  describes  three  strategy  elements  and  their  sub-elements  (  Ac¬ 
quisition  Approach,  Business  Considerations,  and  Product  Support)  in  more  detail.  We  will 
later  use  these  strategy  elements  illustrate  the  strategy  development  method.  For  each  of  these 
three  strategy  elements  and  sub-elements,  we  define  the  element  (or  sub-element),  discuss  the 
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range  of  strategy  choices,  and  provide  examples  of  those  choices.  Then,  using  the  categories 
of  software  characteristics  and  subsequent  strategy  drivers  documented  in  Section  3  (and 
summarized  in  Table  6),  we  present  a  “strategy  profile”  for  each  strategy  element  or  sub¬ 
element.  The  strategy ?  profile  summarizes  the  principle  drivers  that  influence  a  strategy  ele¬ 
ment  and  are  presented  in  Table  7-Table  13. 


Table  6:  Strategy  Driver  Taxonomy 


1.  Software  Criticality 


2.  Acquisition  Environment  Category  Drivers 

a.  Policies  and  Mandates 

b.  Supplier  Availability 

3.  Programmatic  Category  Drivers 

a.  Mission  Needs  and  Scope 

b.  Funding 

i.  Funding  Constraints 

ii.  Funding  Profile 

c.  Schedule 

i.  Schedule  Constraints 

ii.  Urgency 

4.  Organizational  Category  Drivers 

a.  Program  Management  Office  Capabilities 

i.  PMO  Staff  Skills 

ii.  PMO  Staff  Capacity 
Hi.  PMO  Staff  Stability 
iv.  PMO  Process  Focus 

b.  Stakeholders 

i.  Number  and  Diversity 

ii.  Level  of  Engagement  (responsiveness 
and  quality’) 

iii.  Level  of  Agreement 

c.  Supplier  Capability 

i.  Supplier  Staff  Skills 

ii.  Supplier  Staff  Capacity 

iii.  Supplier  Staff  Stability 

iv.  Supplier  Performance  to  Date 


Life  Cycle  Category  Drivers 

a.  Product  Definition  and  Specification 

i.  Requirements  Volatility 

ii.  Requirements  Understanding 

iii.  Quality  Attribute  Definition  Quality 

iv.  Interoperability 

b.  Architecture  and  Design 

i.  Precedence 

ii.  Quality  Attribute  Constraints 

iii.  Technology’  Readiness 

iv.  Legacy  Considerations 

v.  COTS  /  GOTS  /  Reuse 

c.  Verification  and  Test 

i.  Test  Environment  Complexity’ 

ii.  Test  Environment  Availability 

iii.  Number  of  System  Configurations 

d.  Deployment 

i.  Number  of  Sites 

ii.  User  Readiness 

iii.  Maintainer  Readiness 

iv.  Transition  /  Data  Migration 

e.  Maintenance  and  Support 

i.  Number  of  System  Configurations 

ii.  Update  Readiness 

iii.  Support  Duration 

iv.  Re-Competition  Readiness 

v.  Operational  Environment 

vi.  Legacy  Considerations 

vii.  Complexity  of  Data  Rights 

f.  Disposal 
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4.2  Acquisition  Approach  Strategy  Element 

Note:  Some  of  the  information  contained  in  this  section  is  familiar  to  most  experienced  ac¬ 
quisition  PMO  staff.  It  is  reiterated  in  this  report  to  lend  coherence  to  the  authors’  discussion 
of  acquisition  strategy  elements. 


The  Acquisition  Approach  strategy  element  defines  the  approach  the  program  will  use  to 
achieve  full  capability.  Typically,  this  approach  is  either  single  step,  evolution¬ 
ary/incremental,  or  evolutionary/spiral,  as  illustrated  in  Figure  9. 


Single  Step 
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Based  on  AF  Program  Manager  Workshop  presented  by  Mr.  Little 


Figure  9:  Defining  an  Acquisition  Approach 

4.2.1  Single-Step  Approach 

As  the  name  implies,  the  single-step  acquisition  approach  consists  of  the  acquisition  of  a  de¬ 
fined  capability  in  one  increment.  The  delivered  items  provide  100%  of  the  required  capabil¬ 
ity.  To  achieve  this  result,  the  acquirer  must  know  all  of  the  requirements  at  the  start  of  the 
program.  Of  course,  virtually  no  program  proceeds  to  completion  without  the  addition  or 
modification  of  some  requirements;  however,  many  programs  do  start  with  a  firm  definition 
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of  the  goal  and  a  clear  plan  on  how  to  attain  it.  For  these  types  of  programs,  requirements 
changes  are  typically  less  significant. 

The  single-step  acquisition  approach  is  most  appropriate  for  programs  that  have  the  following 
characteristics: 

•  The  capability  is  clearly  defined. 

•  All  of  the  requirements,  including  the  software  requirements,  are  known. 

•  The  technology  to  achieve  the  capability  is  mature. 

•  There  is  no  demand  for  earlier  deployment  of  partial  capability. 

•  There  are  successful  precedents  for  the  program  (i.e.,  other  similar  programs  delivering 
similar  capabilities  have  been  successfully  completed). 

4.2.2  Evolutionary/Incremental  Approach 

The  evolutionary/incremental  acquisition  approach  consists  of  the  acquisition  of  a  defined 
capability  in  multiple,  defined  increments.  Each  increment  provides  a  defined  and  “fieldable” 
capability.  Initial  increments  may  provide  only  a  portion  of  the  total  capability  required  with 
each  increment  adding  more  capability  until  full  capability  is  achieved.  Because  the  capabili¬ 
ties  delivered  in  each  increment  are  pre-defined  during  the  acquisition  planning  process, 
again,  the  acquirer  must  know  all  of  the  requirements  at  the  start  of  the  program. 

The  evolutionary/incremental  acquisition  approach  provides  more  flexibility  than  the  single- 
step  method;  it  enables  the  acquirer  to  address  issues  arising  from  either  technology  maturity 
or  user  needs  for  early  fielding  of  some  capabilities.  This  flexibility  may  come  at  the  expense 
of  more  effort  for  the  acquirer,  more  cost,  and/or  longer  schedules. 

The  evolutionary/incremental  acquisition  approach  is  most  appropriate  for  programs  that 
have  the  following  characteristics: 

•  The  capability  is  clearly  defined. 

•  All  of  the  requirements,  including  the  software  requirements,  are  known. 

•  The  technology  to  achieve  the  capability  in  the  first  increment  is  mature.  The  technolo¬ 
gies  to  achieve  the  capabilities  of  the  later  increments  are  proven,  but  require  further  de¬ 
velopment  before  deployment. 

•  There  is  a  demand  for  earlier  deployment  of  some  partial  capabilities. 

•  There  are  successful  precedents  for  the  program  (i.e.,  other  similar  programs  delivering 
similar  capabilities  have  been  successfully  completed). 

4.2.3  Evolutionary/Spiral  Approach 

The  focus  of  the  evolutionary/spiral  acquisition  approach  is  risk  mitigation  and  risk  reduc¬ 
tion.  This  approach  consists  of  the  acquisition  of  a  defined  capability  in  multiple,  undefined 
increments.  Each  increment  is  often  referred  to  as  a  “spiral.”  At  the  beginning  of  each  in- 
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crement,  the  acquirer  defines  a  “deliverable”  capability  for  that  increment.  Initial  increments 
focus  on  the  aspects  of  the  capability  that  inject  the  greatest  risk  into  the  program.  These  are 
usually  aspects  associated  with  unprecedented  capabilities  and  new,  immature  technologies. 
At  the  end  of  each  increment,  the  results  of  the  increment  are  reviewed  and  a  risk  analysis  of 
the  overall  program  is  performed.  Based  on  this  evaluation,  the  next  increment  is  defined. 
Only  after  these  risky  endeavors  have  been  completed  does  the  acquirer  begin  to  address  the 
more  conventional,  less  risky  aspects  of  the  program.  In  this  manner,  a  “showstopping”  tech¬ 
nological  issue  can  be  identified  early  in  the  program.  This  enables  the  acquirer  to  consider 
alternative  approaches  before  significant  “sunk  costs”  are  incurred. 

To  successfully  execute  an  evolutionary /spiral  acquisition,  the  acquirer  first  needs  a  clear 
understanding  of  the  final  capability  that  is  desired.  This  serves  as  the  target  for  the  success¬ 
ful  acquisition  spirals.  Driven  by  a  risk  analysis  of  the  program,  the  acquirer  identifies  its 
most  significant  risks  and  forms  a  risk  reduction  strategy  to  address  them.  Such  a  strategy  can 
result  in  early  increments  that  involve 

•  evaluating  candidate  system  components  in  the  context  of  the  program 

•  creating  laboratory  or  pre-production  prototypes  of  challenging  components 

•  transitioning  immature  technologies  from  laboratory  to  the  program  environment 
Subsequent  increments  build  on  the  earlier  increments. 

Because  the  capabilities  delivered  in  each  increment  are  defined  at  the  initiation  of  that  in¬ 
crement,  the  acquirer  does  not  need  to  know  all  of  the  system  capabilities  at  the  initiation  of 
the  program.  He  or  she  must  only  know  all  of  the  requirements  for  the  next  increment  at  the 
start  of  that  increment.  Requirements  for  the  next  increment  are  defined  only  during  and  after 
execution  of  the  current  increment. 

In  this  manner,  the  evolutionary/spiral  acquisition  approach  provides  considerably  more 
flexibility  than  the  single-step  method,  enabling  the  acquirer  to  address  high-risk  issues 
within  the  program.  This  flexibility  may  come  at  the  expense  of  more  effort  for  the  acquirer, 
more  cost,  and/or  longer  schedules. 

Key  points  that  are  crucial  to  the  success  of  spiral  acquisition  include 

•  The  acquirer  needs  to  obtain  a  clear  understanding  of  the  final  capability  that  is  needed. 

•  Increments  should  be  risk-driven;  each  increment  should  be  defined  to  reduce  the  highest 
risks  facing  the  program  at  that  time. 

•  At  the  start  of  each  spiral,  the  acquisition  plan  should  be  revisited  to  fully  specify  the  ca¬ 
pabilities  to  be  delivered  in  that  spiral  and  to  tentatively  define  the  future  spirals  leading 
to  final  capability. 

•  Spirals  should  have  well  defined  exit  points. 

The  evolutionary/spiral  acquisition  approach  is  most  appropriate  for  programs  that  have  the 
following  characteristics: 
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•  The  capability  is  not  folly  defined  or  the  stakeholders  have  not  reached  consensus  on  the 
capability  definition. 

•  All  of  the  requirements,  including  the  software  requirements,  are  not  known  (but  re¬ 
quirements  for  the  first  increment  are  known). 

•  All  of  the  technology  needed  to  achieve  the  final  capability  is  not  folly  mature  (but  the 
technology  required  for  the  first  increment  is  mature). 

•  There  may  be  a  demand  for  earlier  deployment  of  some  partial  capabilities. 

•  There  are  not  successful  precedents  for  the  program  (i.e.,  other  similar  programs  have  not 
delivered  similar  capabilities). 

Unlike  many  strategy  elements,  the  Acquisition  Approach  strategy  element  has  a  defined 
range  of  choices  consisting  of  only  the  three  options  presented  above.  As  discussed,  the  key 
program  drivers  influencing  the  choice  of  an  acquisition  approach  is  primarily  influenced  by 
the  completeness  of  the  requirements,  requirements  volatility,  and  the  demand  for  interim 
capability  delivery. 

The  strategic  choices  and  drivers  of  the  Acquisition  Approach  strategy  element  are  summa¬ 
rized  in  the  strategy  profile  shown  in  Table  7. 
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Table  7;  Acquisition  Approach  Strategy  Profile 


Strategy 

Element  Range 

Single-Step 

Evolutionary  /  Incremental 

Evolutionary  /  Spiral 

Principal  driver 

1.  Software  Criticality 

5.  Life  Cycle  Category  Drivers 

influences 

2.  Acquisition  Environment  Cat.  Drivers 

a.  Product  Definition  and 

(from  Table  6) 

a.  Policies  and  Mandates 

Specification 

b.  Supplier  Availability 

i.  Requirements  Volatility 

3.  Programmatic  Category  Drivers 

ii.  Requirements  Understanding 

a.  Mission  Needs  and  Scope 

iii.  Quality’  Attribute  Definition 

b.  Funding 

Quality’ 

i.  Funding  Constraints 

iv.  Interoperability’ 

ii.  Funding  Profile 

b.  Architecture  and  Design 

c.  Schedule 

i.  Precedence 

i.  Schedule  Constraints 

ii.  Quality’  Attribute  Constraints 

ii.  Urgency 

iii.  Technology’  Readiness 

4.  Organizational  Category  Drivers 

iv.  Legacy  Considerations 

a.  Program  Management  Office 
Capabilities 

i.  PMO  Staff  Skills 

ii.  PMO  Staff  Capacity 

Hi.  PMO  Staff  Stability 

b.  Stakeholders 

i.  Number  and  Diversity 

ii.  Level  of  Engagement 
(responsiveness  and  quality) 

iii.  Level  of  Agreement 

v.  COTS  /  GOTS  /  Reuse 

4.3  Business  Considerations:  Competition  Strategy 
Element 

Note:  Some  of  the  information  contained  in  this  section  is  familiar  to  most  experienced  ac¬ 
quisition  PMO  staff.  It  is  reiterated  in  this  report  to  lend  coherence  to  the  authors’  discussion 
of  acquisition  strategy  elements. 

Competition  is  a  key  element  of  the  acquisition  strategy;  it  is  recognized  as  a  means  of 
achieving  the  best  acquisition  value.  The  acquirer  should  also  use  it  as  a  means  of  fostering 
innovation  for  defense  applications.  During  competition,  potential  suppliers  analyze  the 
needs  of  the  acquirer  and  provide  their  best  ideas  to  address  these  needs.  More  competition 
results  in  more  ideas,  some  of  which  may  be  innovative  and  valuable  to  the  acquirer. 
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Competition  options  (strategic  choices)  for  the  acquirer  include 

•  Full  and  Open  Competition 

•  Full  and  Open  Competition  After  Exclusion  of  Sources 

•  Sole  Source  Contracting 

The  default  competition  strategy  is  full  and  open  competition;  however,  Federal  Acquisition 
Regulations  (FAR)  Part  6,  Subpart  6.3 3  defines  the  circumstances  under  which  full  and  open 
competition  may  be  waived.  Full  and  open  competition  may  be  implemented  through  sealed 
bids,  competitive  proposals,  combinations  of  competitive  procedures  (e.g.,  two-step  sealed 
bidding),  or  other  competitive  procedures  as  described  in  FAR  Part  6,  Subpart  6.1. 

Full  and  open  competition  after  exclusion  of  sources  enables  the  acquirer  to  restrict  the  sup¬ 
pliers  who  may  respond  to  the  solicitation.  After  the  exclusion  of  sources,  full  and  open  com¬ 
petition  is  implemented  for  the  remaining  sources.  An  acquirer  may  exclude  potential  sources 
to 

•  establish  or  maintain  alternative  sources  (i.e.,  exclude  bidders  to  provide  opportunity  for 
other  bidders  to  enter  the  market)  (FAR  6.202) 

•  limit  the  bidders  to  Small  Business  Concerns  (FAR  6.203) 

•  execute  the  acquisition  through  the  Small  Business  Administration,  again  limiting  the 
bidders  to  Small  Business  Concerns  (FAR  6.204) 

•  limit  the  bidders  to  FlUBZone  Small  Business  Concerns  (FAR  6.205) 

•  limit  the  bidders  to  Service-Disabled  Veteran-Owned  Small  Business  Concerns  (FAR 
6.206) 

Sole  source  competition  enables  the  acquirer  to  award  a  contract  without  full  and  open  com¬ 
petition.  This  is  permitted  only  under  the  following  extraordinary  conditions: 

•  Only  One  Responsible  Source  and  No  Other  Supplies  or  Services  Will  Satisfy  Agency 
Requirements  (FAR  6.302-1) 

•  Unusual  and  Compelling  Urgency  (FAR  6.302-2) 

•  Industrial  Mobilization;  Engineering,  Developmental,  or  Research  Capability;  or  Expert 
Services  (FAR  6.302-3) 

•  International  Agreement  (FAR  6.302-4) 

•  Authorized  or  Required  by  Statute  (FAR  6.302-5) 

•  National  Security  (FAR  6.302-6) 

•  Public  Interest  (FAR  6.302-7) 


3  Throughout  this  report,  a  number  of  references  are  made  to  the  Federal  Acquisition  Regulations 
(FARs).  Additional  regulations  such  as  the  Defense  Federal  Acquisition  Regulations  Supplement 
(DFARS),  Army  Federal  Acquisition  Regulations  Supplement  (AFARS),  Air  Force  Federal  Acqui¬ 
sition  Regulations  Supplement  (AFFARS),  may  also  apply. 
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The  acquirer  must  define  the  competition  planned  for  all  phases  of  the  program’s  life  cycle 
(e.g.,  Concept  Refinement,  Technology  Development,  System  Development  and  Demonstra¬ 
tion,  Production  and  Deployment,  and  Operations  and  Support).  Different  competition  strate¬ 
gies  may  be  used  in  these  life-cycle  phases.  For  example,  full  and  open  competition  may  be 
used  in  Concept  Refinement  and  Technology  Development  phases,  with  sole  source  contract¬ 
ing  used  for  later  phases.  When  utilizing  evolutionary  (either  incremental  or  spiral)  acquisi¬ 
tion,  the  acquirer  should  evaluate  the  costs  and  the  benefits  of  competing  each  increment. 

Competition  must  also  be  considered  when  contracting  for  support  of  the  product.  If  support 
is  not  contracted  as  part  of  system  development,  the  acquirer  must  ensure  that  appropriate 
technical  data  packages  and  appropriate  data  rights  are  procured  during  the  development 
phase  to  support  competition  and  contracting  of  support. 

Factors  that  influence  the  Competition  strategy  elements  include 

•  available  sources  of  supply 

•  small  business  set-aside  statutes 

•  special  circumstances  (e.g.,  only  one  responsible  source,  urgency,  industrial  mobilization, 
international  agreement,  other  statutory  requirement,  national  security,  public  interest) 

The  strategic  choices  and  drivers  of  the  Competition  strategy  element  are  summarized  in  the 
strategy  profile  shown  in  Table  8. 
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Table  8:  Competition  Strategy  Profile 


Strategy 

Element  Range 

Full  and  Open  Competition 

Full  and  Open  Competition  After  Exclusion  of  Sources 

Sole  Source  Contracting 

Principal  driver 

influences 

(from  Table  6) 

1.  Software  Criticality  5.  Life  Cycle  Category  Drivers 

2.  Acquisition  Environment  Cat.  Drivers  a.  Product  Definition  and 

a.  Policies  and  Mandates  Specification 

b.  Supplier  Availability  i.  Requirements  Volatility’ 

3.  Programmatic  Category  Drivers  ii.  Requirements  Understanding 

a.  Mission  Needs  and  Scope  Hi.  Quality  Attribute  Definition 

b.  Funding  Quality 

i.  Funding  Constraints  iv.  Interoperability 

ii.  Funding  Profile  b.  Architecture  and  Design 

c.  Schedule  i.  Precedence 

i.  Schedule  Constraints  ii.  Quality  Attribute  Constraints 

ii.  Urgency’  iii.  Technology’  Readiness 

4.  Organizational  Category  Drivers  iv.  Legacy  Considerations 

a.  Program  Management  Office  v.  COTS  /  GOTS  /  Reuse 

Capabilities  c.  Verification  and  Test 

i.  PMO  Staff  Skids  i.  Test  Environment  Complexity 

ii.  PMO  Staff  Capacity  ii.  Test  Environment  Availability 

iii.  PMO  Staff  Stability’  iii.  Number  of  System 

Configurations 
d.  Deployment 

i.  Number  of  Sites 

ii.  User  Readiness 

iii.  Maintainer  Readiness 

iv.  Transition  /  Data  Migration 

4.4  Business  Considerations:  Solicitation  Strategy 
Element 

Note:  Some  of  the  information  contained  in  this  section  is  familiar  to  most  experienced  ac¬ 
quisition  PMO  staff.  It  is  reiterated  in  this  report  to  lend  coherence  to  the  authors’  discussion 
of  acquisition  strategy  elements. 

A  solicitation  is  often  the  acquirer’s  first  formal  attempt  to  transmit  his  needs  to  the  constel¬ 
lation  of  potential  suppliers.  It  is  also  often  the  first  time  that  many  suppliers  become  aware 
of  the  proposed  acquisition.  Typical  forms  of  solicitation  include  the  following: 
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•  Invitation  for  Bid  (IFB)  -An  IFB  is  a  solicitation  issued  by  the  acquirer  to  the  commu¬ 
nity  of  potential  bidders.  An  IFB  describes  the  needs  of  the  acquirer  (i.e.,  the  require¬ 
ments)  and  the  criteria  for  the  evaluation  of  the  offerors’  bids  and  proposals.  Contract 
award  is  made  without  negotiation  and  is  based  on  the  lowest  bid.  This  solicitation 
method  is  not  typically  suitable  for  the  acquisition  of  complex,  custom-built,  software  in¬ 
tensive  systems. 

•  Request  for  Proposal  (RFP)  -  An  RFP  is  a  solicitation  issued  by  the  acquirer  to  the 
community  of  potential  bidders.  An  RFP  describes  the  needs  of  the  acquirer  (i.e.,  the  re¬ 
quirements)  and  the  criteria  for  the  evaluation  of  the  offerors’  bids  and  proposals.  Evalua¬ 
tion  criteria  typically  include  factors  such  as  price,  schedule,  technical  merit,  manage¬ 
ment  capability,  risk,  and  so  on.  Contract  award  is  based  upon  evaluation  of  specified 
factors.  Negotiations  can  be  conducted  with  offerors. 

An  RFP  can  be  issued  in  combination  with  a  Statement  of  Work  (SOW)  or  Statement  of 
Objectives  (SOO).  A  SOW  provides  a  detailed  definition  of  the  scope  of  the  contract; 
whereas,  a  SOO  provides  a  more  general  statement  of  the  objectives  of  the  contract.  For 
example,  a  SOW  may  be  used  to  define  the  requirements  for  an  air-to-air  missile.  A  SOO 
may  be  used  to  define  the  objectives  of  destroying  airborne  targets  of  defined  characteris¬ 
tics  within  a  defined  range.  These  objectives  could  be  met  with  a  missile,  artillery,  di¬ 
rected  energy  weapons,  etc. 

For  complex  systems  and/or  large  programs,  it  is  common  to  issue  an  RFP  several  times 
in  a  draft  form  as  a  means  of  soliciting  input  and  feedback  from  the  community  of  poten¬ 
tial  bidders.  Not  only  does  this  process  improve  the  quality  of  the  RFP,  it  also  helps  solid¬ 
ify  the  acquirers’  concepts  of  what  is  required,  and  helps  the  potential  bidders  to  form  a 
more  complete  understanding  of  the  acquirers  needs. 

•  Request  for  Quotation  (RFQ)  -  An  RFQ  is  a  request  for  information  made  by  the  ac¬ 
quirer  to  the  community  of  prospective  suppliers.  It  is  typically  used  for  planning  pur¬ 
poses.  It  is  not  a  commitment  to  procure  the  requested  product.  As  a  result  of  the  re¬ 
sponse  to  an  RFQ,  the  acquirer  can  refine  the  product  requirements  before  issuing  an  IFB 
or  RFP. 

•  Request  for  Information  (RFI)  -  An  RFI  is  similar  to  an  RFQ  in  that  it  is  used  to  solicit 
knowledge  and  information  from  the  community  of  prospective  suppliers  in  the  earliest 
phases  of  an  acquisition.  Whereas  the  RFQ  is  seeking  primarily  cost  and  schedule  infor¬ 
mation  to  be  used  for  acquisition  planning  purposes,  the  RFI  is  often  seeking  technical 
information  regarding  program  feasibility,  technology  maturity,  bidder  capabilities,  etc. 

The  strategic  choices  and  drivers  of  the  Solicitation  strategy  element  are  summarized  in  the 

strategy  profile  shown  in  Table  9. 
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Table  9:  Solicitation  Strategy  Profile 


Strategy 

Element  Range 

Invitation  for  Bid  (IFB) 

Request  for  Proposal  (RFP)  with  SOW 

Request  for  Proposal  (RFP)  with  SOO 

Request  for  Quotation  (RFQ) 

Principal  driver 

1. 

Software  Criticality 

5.  Life  Cycle  Category  Drivers 

influences 

2. 

Acquisition  Environment  Cat.  Drivers 

a.  Product  Definition  and 

(from  Table  6) 

a.  Policies  and  Mandates 

Specification 

b.  Supplier  Availability 

i.  Requirements  Volatility 

3. 

Programmatic  Category  Drivers 

ii.  Requirements  Understanding 

a.  Mission  Needs  and  Scope 

iii.  Quality  Attribute  Definition 

b.  Funding 

Quality 

i.  Funding  Constraints 

iv.  Interoperability 

c.  Schedule 

b.  Architecture  and  Design 

i.  Schedule  Constraints 

i.  Precedence 

ii.  Urgency 

ii.  Quality  Attribute  Constraints 

4. 

Organizational  Category  Drivers 

iii.  Technology’  Readiness 

a.  Program  Management  Office 

iv.  Legacy  Considerations 

Capabilities 

v.  COTS  /  GOTS  /  Reuse 

i.  PMO  Staff  Skills 

ii.  PMO  Staff  Capacity 

Hi.  PMO  Staff  Stability 

4.5  Business  Considerations:  Contract  Approach 
Strategy  Element 

Note:  Some  of  the  information  contained  in  this  section  is  familiar  to  most  experienced  ac¬ 
quisition  PMO  staff.  It  is  reiterated  in  this  report  to  lend  coherence  to  the  authors’  discussion 
of  acquisition  strategy  elements. 

A  contract  defines  the  relationship  between  the  acquirer  and  the  supplier.  The  acquirer  must 
choose  the  type  of  contract  to  issue  for  each  acquisition.  Contract  types  can  be  categorized  as 

1.  Fixed-Price  Contracts  (FAR  16.2) 

2.  Cost  Contracts  (FAR  16.302) 

3.  Incentive  Contracts  (FAR  16.4) 

4.  Indefinite-Delivery  Contracts  (FAR  16.5) 

5.  Time-and-Materials,  Labor-Hour,  and  Letter  Contracts  (FAR  16.6) 

6.  Agreements  (FAR  16.7) 
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The  first  three  of  contract  categories  are  most  applicable  to  acquisition  of  developmental 
items  and  are  addressed  in  this  section.  This  section  will  begin  with  a  definition  of  the  vari¬ 
ous  contract  types  and  then  addresses  when  a  particular  contract  type  might  be  applicable. 


4.5.1  Fixed-Price  Contracts  (FAR  16.2) 

Fixed-price  contracts  provide  a  defined  deliverable  in  return  for  a  defined  price  and  are  usu¬ 
ally  considered  the  default  contract  type.  Six  variations  of  fixed-price  contracts  are  discussed 
below. 

•  For  a  Firm-Fixed-Price  (FFP)  Contract  (FAR  16.202),  the  price  is  not  subject  to  any  ad¬ 
justment  on  the  basis  of  the  contractor’s  costs.  FFP  contracts  are  most  appropriate  for  ac¬ 
quisition  of  commercial  items.  They  are  also  appropriate  for  acquisition  of  developmental 
items  with  stable  and  detailed  specifications  if  the  acquirer  can  identify  a  fair  and  reason¬ 
able  price  using  methods  such  as  price  competition,  prior  purchase  history,  and  so  on. 

•  A  Fixed-Price  Contract  with  Economic  Price  Adjustment  (FAR  16.203)  also  establishes  a 
fixed  price;  however,  it  may  be  adjusted  (up  or  down)  in  response  to  specified  contingen¬ 
cies  such  as 

-  changes  in  the  market  price  of  specific  items  used  in  the  execution  of  the  contract 

-  changes  in  specified  costs  of  labor  or  material  experienced  by  the  contractor 

-  changes  in  specified  labor  or  material  cost  standards  or  indexes 

This  contract  type  is  used  when  there  is  significant  doubt  regarding  the  stability  of  the 
economic  conditions  throughout  the  duration  of  the  contract. 

•  A  Fixed-Price  Contract  with  Prospective  Price  Redetermination  (FAR  16.205)  provides  a 
firm,  fixed  price  for  an  initial  period  of  the  contract  and  a  plan  for  redetermination,  at  de¬ 
fined  times  during  contract  execution,  of  the  price  for  subsequent  periods.  The  contract 
typically  includes  a  cap  on  future  price  determinations.  This  type  of  contract  is  used 
when  the  acquirer  can  negotiate  a  fair  and  reasonable  FFP  for  the  initial  period,  but  not 
for  later  periods. 

•  A  Fixed-Ceiling-Price  Contract  with  Retroactive  Price  Redetermination  (FAR  16.206) 
establishes  a  fixed  ceiling  price  and  provides  for  price  redetermination  within  the  ceiling 
after  contract  completion.  This  type  of  contract  is  used  for  small  research  and  develop¬ 
ment  contracts  if  a  fair  and  reasonable  firm,  fixed  price  cannot  be  established. 

•  A  Firm-Fixed-Price,  Level-of-Effort  Term  Contract  (FAR  16.207)  provides  a  specified 
level  of  effort,  over  a  stated  period  of  time,  on  work  that  can  only  be  defined  in  general 
terms,  in  return  for  a  fixed  dollar  value.  This  type  of  contract  is  typically  used  for  study 
of  a  specific  topic. 

4.5.2  Cost-Reimbursement  Contracts  (FAR  16.3) 

Cost-reimbursement  types  of  contracts  pay  for  costs  incurred  by  the  supplier  during  the  exe¬ 
cution  of  the  contract.  These  contracts  establish  a  cost  ceiling  and  obligate  funds  based  upon 
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an  estimate  of  total  cost.  They  are  appropriate  for  contracts  where  significant  uncertainties 
preclude  the  possibility  of  accurately  estimating  expected  costs.  Three  variations  of  cost- 
reimbursement  contracts  are  discussed  below. 

•  A  Cost  Contract  (FAR  16.302)  provides  reimbursement  of  the  contractors’  costs,  but  the 
contractor  receives  no  fee.  This  type  of  contract  is  usually  applied  to  research  work  per¬ 
formed  by  non-profit  organizations. 

•  A  Cost-Sharing  Contract  (FAR  16.303)  provides  reimbursement  of  only  a  portion  of  the 
contractors’  costs.  Again,  the  contractor  receives  no  fee.  This  type  of  contract  is  used 
when  the  contractor  is  willing  to  absorb  some  of  the  contract  costs  in  the  expectation  of 
some  future  benefit  (e.g.,  entry  into  a  new  market,  development  of  a  new  technology,  im¬ 
proved  competitive  position  for  future  acquisitions). 

•  A  Cost-Plus-Fixed-Fee  Contract  (FAR  16.306)  provides  reimbursement  of  the  contrac¬ 
tors’  costs.  Additionally,  the  contractor  receives  a  fixed  fee  determined  at  contract  award. 
This  type  of  contract  is  typically  used  on  small  contracts  for  which  expected  costs  cannot 
be  accurately  estimated. 

4.5.3  Incentive  Contracts  (FAR  16.4) 

Incentive  contracts  reimburse  the  costs  of  the  contractor  and  also  provide  a  fee.  Like  cost- 
reimbursement  contracts,  contract  performance  targets  (e.g.  cost  targets,  schedule  targets, 
technical  performance  targets)  are  defined.  Unlike  the  cost  reimbursement  contracts,  the  fee 
is  structured  in  a  way  to  incentivize  the  contractor  to  meet  these  targets.  Four  variations  of 
incentive  contracts  are  discussed  below: 

•  For  a  Fixed-Price  Incentive  Contract  (FAR  16.403),  the  acquirer  and  contractor  negotiate 
a  target  cost,  a  fee  schedule,  and  a  ceiling  price.  The  fee  varies  inversely  to  the  relation¬ 
ship  between  the  actual  performance  and  the  performance  targets  (e.g.,  greater  cost  => 
smaller  fee,  later  delivery  =>  smaller  fee,  less  technical  performance  =>  smaller  fee).  This 
type  of  contract  is  used  where  the  level  of  cost  uncertainty  is  too  great  for  a  fixed-price 
contract;  however,  the  costs  can  be  bounded  sufficiently  to  establish  a  ceiling  acceptable 
to  both  parties.  The  incentive  fee  then  motivates  the  contractor  to  perform  efficiently 
within  this  boundary. 

•  A  Fixed-Price  Contract  with  Award  Fees  (FAR  16.404)  is  a  fixed-price  contract.  Award 
fees  are  added  to  the  contract  value  to  motivate  contractor  performance.  Award  fees  are 
paid  based  upon  the  acquirer’s  evaluation  of  the  contractor’s  performance.  This  type  of 
contract  is  used  when  objective  contractor  performance  measures  are  not  practical. 

•  A  Cost-Plus-Incentive-Fee  Contract  (FAR  16.304)  provides  reimbursement  of  the  con¬ 
tractor’s  costs.  Additionally,  a  negotiated  fee  is  paid.  The  acquirer  and  contractor  negoti¬ 
ate  a  target  cost  and  a  fee  schedule.  The  resulting  fee  is  adjusted  based  upon  the  relation¬ 
ship  between  actual  and  target  costs.  This  type  of  contract  is  used  when  the  level  of  cost 
uncertainty  is  too  great  for  a  fixed-price  contract.  The  incentive  fee  then  motivates  the 
contractor  to  perform  efficiently. 
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•  A  Cost-Plus-Award-Fee  Contract  (FAR  16.405-2)  provides  reimbursement  of  the  contrac¬ 
tor’s  costs,  and  a  base  fee,  negotiated  at  the  time  of  contract  award.  Additionally,  through 
excellent  performance  (as  evaluated  by  the  acquirer),  the  contractor  may  earn  additional 
award  fees.  This  type  of  contract  is  used  when  the  level  of  cost  uncertainty  is  too  great 
for  a  fixed-price  contract,  and  it  is  impractical  to  establish  pre-defmed  incentive  targets. 

The  use  of  incentive  fees  and  the  use  of  award  fees  are  not  mutually  exclusive,  and  may  be 

used  simultaneously.  Contract  term  may  also  be  used  as  an  incentive,  providing  extensions  or 

reductions  in  the  term  of  the  contract  based  upon  contractor  performance. 

4.5.4  Selecting  a  Contract  Type 

When  selecting  a  contract  type,  the  acquirer  must  consider  several  factors  including 

•  Competition  -  Competition  among  responsible  suppliers  typically  results  in  realistic  pric¬ 
ing,  enabling  the  use  of  fixed-price  contracts. 

•  Complexity  -  Complex  products  are  inherently  more  risky  than  simpler  ones.  The  con¬ 
tract  type  plays  a  role  in  defining  the  level  of  risk-sharing  between  the  acquirer  and  the 
contractor. 

•  Acquisition  Size  -  The  size  of  the  acquisition  as  measured  by  total  cost  or  total  effort,  has 
a  direct  bearing  on  the  choice  of  a  contract  type,  in  that  some  contract  types  are  not  ap¬ 
propriate  for  larger  acquisitions.  Additionally,  larger  acquisitions  are  inherently  more 
risky  than  smaller  acquisitions.  The  contract  type  plays  a  role  in  defining  the  level  of 
risk-sharing  between  the  acquirer  and  the  contractor. 

•  Requirement  Stability  -  Fixed-price  contracts  are  appropriate  only  when  the  product  re¬ 
quirements  are  complete  and  stable  at  the  time  of  solicitation.  Incomplete  requirements  or 
requirements  volatility  add  uncertainty  that  is  unacceptable  for  fixed-price  contracts. 

•  Cost  Estimation  -  The  ability  of  both  the  acquirer  and  the  contractor  to  accurately  predict 
contract  costs  is  crucial  to  the  selection  of  a  contract  type.  Without  accurate  cost  esti¬ 
mates,  fixed-price  contracts  will  not  be  practical.  Cost  estimation  accuracy  derives  from  a 
clear  definition  of  requirements  and  experience  with  similar  systems 

•  Contractor  Capabilities  -  The  contractor’s  technical  capabilities  and  performance  history 
informs  the  degree  of  risk  that  the  acquirer  is  willing  to  share.  The  contractor’s  financial 
security  influences  decisions  on  cost  sharing,  payments  periods,  and  more.  The  suffi¬ 
ciency  of  the  contractor’s  accounting  system  is  a  gating  factor  for  the  use  of  a  cost  reim¬ 
bursable  contract. 

•  Acquirer  Capabilities  -  Some  contract  types  are  more  difficult  to  administer  than  others. 
The  ability  of  the  acquirer  to  address  these  challenges  influences  type  of  contract  that  is 
selected. 

The  strategic  choices  and  drivers  of  the  Contract  Approach  strategy  element  are  summarized 

in  the  strategy  profile  shown  in  Table  10. 


CMU/SEI-2006-TR-002 


59 


Table  10:  Contract  Type  Strategy  Profile 


Strategy 

Element  Range 

Fixed-Price  Contracts 

•  Firm-Fixed-Price  (FFP)  Contracts 

•  Fixed-Price  Contract  with  Economic  Price  Adjustment 

•  Fixed-Price  Contract  with  Prospective  Price  Redetermination 

•  Fixed-Ceiling-Price  Contracts  with  Retroactive  Price  Redetermination 

•  Firm-Fixed-Price,  Level-of-Effort  Term  Contract 

Cost  Contracts 

•  Cost  Contract 

•  Cost-Sharing  Contract 

•  Cost-Plus-Fixed-Fee  Contract 

Incentive  Contracts 

•  Fixed-Price  Incentive  Contract 

•  Fixed-Price  Contract  With  Award  Fees 

•  Cost-Plus-Incentive-Fee  Contract 

•  Cost-Plus-Award-Fee  Contract 

Principal  driver 

1.  Software  Criticality 

c.  Supplier  Capability 

influences 

2.  Acquisition  Environment  Cat.  Drivers 

i.  Supplier  Staff  Skills 

(from  Table  6) 

a.  Policies  and  Mandates 

ii.  Supplier  Sta  ff  Capacity 

b.  Supplier  Availability 

iii.  Supplier  Staff  Stability 

3.  Programmatic  Category  Drivers 

iv.  Supplier  Performance  to 

a.  Mission  Needs  and  Scope 

Date 

b.  Funding 

5.  Life  Cycle  Category  Drivers 

i.  Funding  Constraints 

a.  Product  Definition  and 

ii.  Funding  Prof  le 

Specification 

c.  Schedule 

i.  Requirements  Volatility 

i.  Schedule  Constraints 

ii.  Requirements  Understanding 

ii.  Urgency 

iv.  Interoperability 

4.  Organizational  Category  Drivers 

b.  Architecture  and  Design 

a.  Program  Management  Office 

i.  Precedence 

Capabilities 

ii.  Quality  Attribute  Constraints 

i.  PMO  Staff  Skills 

iii.  Technology  Readiness 

ii.  PMO  Staff  Capacity 

iv.  Legacy  Considerations 

Hi.  PMO  Staff  Stability 

b.  Stakeholders 

i.  Number  and  Diversity 

ii.  Level  of  Engagement 
(responsiveness  and  quality) 

iii.  Level  of  Agreement 

v.  COTS  /  GOTS  /  Reuse 

4.6  Information  Assurance  Strategy  Element 

The  DoD  defines  information  assurance  (IA)  as  “measures  that  protect  and  defend  informa¬ 
tion  and  information  systems  by  ensuring  their  availability,  integrity,  authentication,  confi- 
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dentiality,  and  non-repudiation.  This  includes  providing  for  the  restoration  of  information 
systems  by  incorporating  protection,  detection,  and  reaction  capabilities  [DAU  04].”  Systems 
today  are  subject  to  threats  throughout  their  life  cycle.  The  goal  of  IA  and  cyber-security  in 
general  is  to  create  and  maintain  a  secure  system  through  a  combination  of  secure  design 
techniques,  secure  operating  procedures,  and  secure  support  procedures.  Security  is  not  a  sys¬ 
tem  attribute  that  is  added  to  a  system.  It  is  an  attribute  that  is  part  of  the  system  definition 
and  impacts  nearly  all  aspects  of  the  system  design.  Just  a  few  of  the  security  challenges 
faced  by  networked  systems  include: 

•  Inclusion  of  malicious  code  during  development  and/or  support  -  In  either  networked  or 
independent  systems,  developers  or  maintainers  may  introduce  malicious  code  during  the 
development  or  support  phase  of  the  system  [GAO  04b].  At  some  future  time,  or  in  re¬ 
sponse  to  some  pre-defmed  stimulus,  such  code  may  cause  system  failure,  alter  system 
operation  in  some  undesirable  manner,  compromise  system  data  security  or  integrity,  or 
all  of  the  above. 

•  Vulnerability  to  unauthorized  access  -  Systems  are  often  intended  for  use  by  a  restricted 
set  of  users.  Without  proper  security  measures  (e.g.,  user  authentication,  access  controls, 
network  monitoring),  both  networked  and  independent  systems  are  subject  to  unauthor¬ 
ized  system  access. 

•  Vulnerability  to  network  attacks  -  Systems  are  connected  to  networks  to  provide  access 
to  multiple  users  at  multiple  locations.  This  connection  enables  the  possibility  of  system 
attack  via  the  network.  A  denial-of-service  attack  can  compromise  system  performance 
by  overwhelming  the  network  with  traffic,  thereby  denying  access  to  the  system  via  the 
network.  Other  forms  of  attack  include  exploitation  of  security  weaknesses  to  gain  unau¬ 
thorized  access  to  the  system  with  the  intent  of  unauthorized  use,  system  software  de¬ 
struction,  system  data  extraction,  or  system  data  corruption.  Such  attacks  can  be  targeted 
at  specific  systems  (i.e.,  hackers  breaking  into  a  target  system)  or  can  be  more  ambiguous 
(i.e.,  infection  by  a  network-carried  virus,  worm,  Trojan  horse). 

IA  is  a  complex  task  that  impacts  all  phases  of  a  system’s  life  cycle.4  Many  techniques  exist 
to  enhance  system  security  and  many  strategy  elements  interact  with  IA  needs.  A  brief  sam¬ 
pling  of  the  IA  strategy  sub-elements  includes: 

1.  supplier  assurance  as  a  means  of  preventing  of  malicious  code  insertion  during  devel¬ 
opment  or  support 

2.  choice  of  communication  network  technology  (e.g.,  open  Internet,  secure  socket  layer, 
Secret  Internet  Protocol  Router  Network  [SIPRNet]) 

3.  means  of  authenticating  users  and  maintainers 

4.  data  encryption  usage 


4  For  additional  information,  visit  the  Build  Security  In  Web  site  at 
https://buildseciirityin.us-cert.gov/daisy/bsi/home.html. 
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As  an  example,  the  subsections  that  follow  discuss  the  first  of  these  strategy  sub-elements 
relating  to  IA. 


4.6.1  Supplier  Assurance  for  Prevention  of  Malicious  Code 
Insertion  During  Development 

One  method  of  reducing  the  chances  of  malicious  code  being  inserted  into  a  system  during 
development  or  maintenance  involves  careful  code  inspections  during  peer  reviews  to  iden¬ 
tify  unauthorized  content,  coupled  with  strict  configuration  management  to  ensure  that  deliv¬ 
ered  code  consists  only  of  known  elements.  Another  method  involves  analyzing  source  code 
and  relating  it  to  system  requirements  to  identify  the  inclusion  of  unauthorized  code  (i.e., 
code  that  does  not  directly  support  the  system  requirements).  Both  of  these  methods  are  only 
as  effective  as  the  vigilance  of  the  parties  tasked  with  ensuring  the  code  quality;  thus,  each 
method  depends  heavily  on  the  exercise  of  care  in  the  selection  of  developers  and  maintain- 
ers.  This  is  the  strategy  element  that  we  address  here. 

The  software  in  a  newly  developed  system  is  often  an  amalgamation  of 

•  software  elements  developed  by  the  system  developer 

•  software  elements  developed  by  the  system  developer’s  suppliers  and/or  subcontractors 

•  software  elements  reused  from  previous  efforts  of  the  system  developer  and/or  subcon¬ 
tractors 

•  COTS,  GOTS  and/or  Free/Open  Source  (F/OS)  software  elements 

Each  of  these  elements  introduces  its  own  risks  into  the  system.  A  strategy  for  addressing 
these  issues  centers  on  the  concept  “Be  careful  who  you  buy  from.”  The  more  secure  your 
source  is,  the  less  your  product  is  likely  to  be  affected  by  this  risk.  Strategies  for  understand¬ 
ing  and  managing  the  acquisition  of  a  system  that  is  critical  to  national  security  might  include 

1.  Use  COTS  and  F/OS  products  freely 

2.  Evaluate  COTS  and  F/OS  products  before  incoiporation 

3.  Use  COTS  and  F/OS  products  only  from  trusted  sources 

4.  No  COTS  or  F/OS  products;  all  software  developed  by  U.S.  sources 

5.  No  COTS  or  F/OS  products;  all  software  developed  by  cleared  U.S.  sources 

4.6. 1.1  Use  COTS  and  F/OS  Products  Freely 

COTS,  GOTS,  and  F/OS  software  elements  pose  a  unique  challenge  to  security  because  the 
developer  may  have  very  little  insight  into  the  internal  workings  of  these  elements.  What  se¬ 
curity  measures  have  been  designed  into  the  product?  How  have  they  been  verified?  Has  the 
developer  incorporated  “back  doors”  and  information-harvesting  processes  into  the  product 
for  his  own  purposes?  These  are  questions  that  may  be  unanswerable.  As  such,  liberal  use  of 
COTS,  GOTS  and  F/OS  products  may  do  little  to  mitigate  information  assurance  risks. 
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4.6. 1.2  Evaluate  COTS  and  F/OS  Products  Before  Incorporation 


It  may  be  possible  to  evaluate  the  security  features  of  COTS,  GOTS  and  F/OS  components 
prior  to  their  incoiporation  into  the  system.  This  usually  requires  the  cooperation  of  the  sup¬ 
plier  and,  at  a  minimum,  access  to  the  component’s  source  code.  This  strategy  primarily  in¬ 
volves  auditing  the  security  practices  claimed  by  the  COTS/GOTS  supplier  (this  may  not  be 
possible  with  F/OS  components  due  to  their  public  nature),  evaluating  their  suitability  for  the 
system,  and  evaluating  the  consistency  of  application  by  the  supplier.  You  can  also  consider  a 
technical  evaluation  of  the  components  themselves,  although  attempting  to  “test-in”  security 
in  this  manner  rather  than  designing  it  into  the  product  is  difficult  and  not  highly  effective. 
Another  tactic  is  to  examine  security  advisories,  bulletins,  and  alerts  to  determine  a  product’s 
history  of  vulnerabilities. 

4.6. 1.3  Use  COTS  and  F/OS  Products  Only  from  Trusted  Sources 

It  may  be  possible  to  evaluate  the  trustworthiness  of  COTS,  GOTS,  and  F/OS  component 
suppliers  prior  to  incorporation  of  their  products  into  the  system.  While  no  widely  recognized 
certification  process  currently  exists,  several  DoD  initiatives  are  working  in  this  direction. 

For  example,  the  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Lo¬ 
gistics)  is  presently  working  with  the  National  Defense  Industrial  Association  Systems  As¬ 
surance  Committee  to  produce  a  guidebook  on  systems  assurance  that  includes  elements  of 
supplier  assurance.  While  there  is  no  formal  certification  for  suppliers  as  yet,  it  is  possible  to 
select  individual  COTS  components  that  have  been  certified  using  standards  such  as  Com¬ 
mon  Criteria  and  The  Common  Criteria  Evaluation  Validation  Scheme. 5 

4.6. 1.4  No  Use  of  COTS  or  F/OS  Products;  All  Software  Developed  by 
U.S.  Sources 

Avoiding  COTS,  GOTS,  and  F/OS  components  eliminates  the  uncertainty  associated  with 
these  types  of  products.  Of  course,  this  leaves  only  the  alternative  of  using  developmental 
items  in  the  construction  of  the  system,  a  practice  that  introduces  a  new  set  of  risks.  For  soft¬ 
ware  elements  developed  by  the  system  developer  and/or  subcontractors,  malicious  employ¬ 
ees  may  introduce  code  that  compromises  system  security.  Opportunities  for  personal  gain 
(i.e.,  blackmail  of  the  developer  or  user)  and  revenge  in  response  to  an  employee  grievance 
are  among  the  common  motivations  for  this  activity.  If  the  system  is  critical  to  national  secu¬ 
rity,  allegiance  to  a  foreign  power  may  also  be  a  motivation.  It  should  also  be  noted  that  not 
all  security  weaknesses  are  a  result  of  malice.  During  development,  developers  may  incorpo¬ 
rate  various  shortcuts  around  security  features  to  simplify  their  work.  Failure  to  remove  these 
shortcuts  completely  from  the  delivered  code  may  also  manifest  itself  in  exploitable  security 
weaknesses.  Another  important  consideration  is  the  development  tools  used;  it  may  be  possi- 


For  more  information,  visit  the  Common  Criteria  portal  at  http://www.commoncriteriaportal.org/ 
and  the  Common  Criteria  Evaluation  Validation  Scheme  Web  site  at 
http://niap.bahialab.com/cc-scheme/. 
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ble  for  developers  using  commercial  tools  to  unwittingly  embed  vulnerabilities  into  the  de¬ 
veloped  code. 


The  reuse  of  software  elements  from  previous  systems  poses  similar  issues,  as  the  same  secu¬ 
rity  weaknesses  may  plague  these  elements.  Additionally,  reused  components  may  include 
features  not  intended  for  use  in  this  system  that  result  in  an  exploitable  security  weakness. 

4.6. 1.5  No  Use  of  COTS  or  F/OS  Products;  All  Software  Developed  by 
Cleared  U.S.  Sources 

Much  like  the  previous  strategy  choice,  this  choice  focuses  on  the  reliability  of  the  develop¬ 
ers.  Using  only  U.S.  sources  with  DoD  security  clearances  provide  further  reinforcement  of 
this  reliability.  While  the  possibility  of  malicious  intent  motivated  by  greed  or  national  dis¬ 
loyalty  still  exists,  the  fact  that  the  organization  and  the  staff  have  been  investigated  and 
cleared  provides  some  reduction  in  this  risk. 

4.6.2  Selecting  a  Strategy  for  Malicious  Code  Prevention 

The  strategic  choices  and  drivers  of  the  Supplier  Assurance  strategy  sub-element  are  summa¬ 
rized  in  the  strategy  profile  shown  in  Table  11. 
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Table  11:  Supplier  Assurance  Strategy  Profile 


Strategy 

Element  Range 

Free  use  of  COTS  and  F/OS  products 

Evaluate  COTS  and  F/OS  products  before  incorporation 

Use  COTS  and  F/OS  products  only  from  trusted  sources 

No  COTS  or  F/OS  products;  all  software  developed  by  US  sources 

No  COTS  or  F/OS  products;  all  software  developed  by  cleared  U.S.  sources 

1.  Software  Criticality 

5.  Life  Cycle  Category  Drivers 

2.  Acquisition  Environment  Category 

a.  Product  Definition  and  Specification 

Drivers 

i.  Requirements  Volatility 

a.  Policies  and  Mandates 

ii.  Requirements  Understanding 

b.  Supplier  Availability 

iii.  Quality  Attribute  Definition 

3.  Programmatic  Category  Drivers 

Quality 

a.  Mission  Needs  and  Scope 

iv.  Interoperability 

b.  Funding 

b.  Architecture  and  Design 

i.  Funding  Constraints 

i.  Precedence 

ii.  Funding  Profile 

ii.  Quality  Attribute  Constraints 

c.  Schedule 

iii.  Technology’  Readiness 

i.  Schedule  Constraints 

iv.  Legacy  Considerations 

ii.  Urgency 

v.  COTS  /  GOTS  /  Reuse 

4.  Organizational  Category  Drivers 

c.  Verification  and  Test 

a.  Program  Management  Office 

i.  Test  Environment  Complexity 

Principal 

Capabilities 

ii.  Test  Environment  Availability 

driver 

i.  PMO  Staff  Skills 

iii.  Number  of  System  Configurations 

influences 

ii.  PMO  Staff  Capacity 

d.  Deployment 

(from  Table  6) 

iii.  PMO  Staff  Stability 

i.  Number  of  Sites 

b.  Stakeholders 

ii.  User  Readiness 

i.  Number  and  Diversity 

iii.  Maintainer  Readiness 

ii.  Level  of  Engagement  (respon¬ 

iv.  Transition  /  Data  Migration 

siveness  and  quality) 

iii.  Level  of  Agreement 

e.  Maintenance  and  Support 

c.  Supplier  Capability 

i.  Number  of  System 

i.  Supplier  Staff  Skills 

Configurations 

ii.  Supplier  Staff  Capacity 

ii.  Update  Readiness 

iii.  Supplier  Staff  Stability 

iii.  Support  Duration 

iv.  Supplier  Performance  to  Date 

iv.  Re-Competition  Readiness 

v.  Operational  Environment 

vi.  Legacy  Considerations 

vii.  Complexity  of  Data  Rights 

f.  Disposal 
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4.7  Product  Support  Strategy:  Training  Strategy 
Element 

Note:  Some  of  the  information  contained  in  this  section  is  familiar  to  most  experienced  ac¬ 
quisition  PMO  staff.  It  is  reiterated  in  this  report  to  lend  coherence  to  the  authors’  discussion 
of  acquisition  strategy  elements. 

In  preparation  for  a  successful  product  deployment,  the  acquirer  must  provide  training  for 
personnel  engaged  with  the  product  including  users,  installers,  operators,  and  maintainers. 

The  acquirer  must  also  address  training  needs  that  will  arise  later  in  the  program,  long  after 
the  acquisition  program  office  has  been  disbanded.  Who  will  train  the  fifth,  or  tenth,  or  one 
hundredth  rotation  of  operators  and  maintainers?  Addressing  this  need  often  results  in  the 
creation  of  a  training  plan  and  training  materials  to  “train  the  trainers.” 

Regardless  of  the  audience  for  training,  several  training  strategies  are  available  to  the  ac¬ 
quirer.  Candidate  strategies  include 

•  Self-Training  -  Self-guided  tutorials  (in  either  paper  or  electronic  format)  provide  the 
trainee  with  the  needed  information  to  perform  his  or  her  task.  This  kind  of  training,  the 
equivalent  of  reading  the  installation  manual,  user’s  manual,  or  service  manual  for  the 
product,  is  only  appropriate  for  products  that  are  very  easy  to  install,  operate,  and  main¬ 
tain.  It  can  be  effective  in  cases  where  the  product  installation,  operation,  and  mainte¬ 
nance  are  highly  intuitive  and  performed  by  highly  capable  staff.  While  this  option  is 
usually  the  least  effective  type  of  training,  it  is  also  the  least  costly. 

•  Computer-Based  Training  (CBT)  -  Like  self-training,  CBT  is  self-paced.  While  both  oc¬ 
cur  without  an  instructor,  unlike  self-training,  CBT  provides  structure,  guidance,  and 
feedback  to  the  trainee.  CBT  can  be  designed  using  proven  instructional  design  methods 
that  establish  learning  objectives,  provide  instruction  to  support  those  objectives,  and  ver¬ 
ify  the  trainee’s  attainment  of  those  objectives.  Typically,  the  CBT  system  provides  in¬ 
struction  in  accordance  with  a  defined  curriculum  using  various  presentation  methods 
(e.g.,  text,  graphics,  animation,  audio,  video)  via  a  computer’s  multimedia  capabilities. 
The  trainee  controls  the  pace  of  the  presentation,  with  the  ability  to  repeat  and  review 
previously  shown  information.  CBT  often  includes  exercises  to  provide  guided  practice 
and  aid  the  trainee’s  absoiption  of  knowledge.  Throughout  the  instruction,  CBT  tests  the 
knowledge  of  the  trainee,  provides  feedback,  and  adjusts  the  remaining  path  of  the  in¬ 
structional  sequence  to  remedy  any  knowledge  deficiencies. 

CBT  can  be  highly  effective  for  all  but  the  most  complex  products.  It  is  highly  economi¬ 
cal  to  deliver  courses  in  this  manner;  however,  course  development  costs  can  be  very 
high. 

•  Distance  Learning  -  When  trainees  are  dispersed  over  a  wide  geographic  area,  distance 
learning  (DL)  may  be  a  suitable  training  option.  Although  many  definitions  of  distance 
learning  exist,  most  agree  that  key  attributes  include 
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a.  the  separation  of  teacher  and  learner  in  space  and/or  time 

b.  the  volitional  control  of  learning  by  the  student  rather  than  the  distant  instructor, 
and 

c.  noncontiguous  communication  between  student  and  teacher,  mediated  by  print  or 
some  form  of  technology  [Sherry  96] 

DL  uses  technology  to  bridge  the  gap  created  by  a  lack  of  immediate  and  personal  com¬ 
munication  between  instructors  and  trainees.  Like  normal  classroom  instruction,  DL  is 
governed  by  a  curriculum,  with  instructors  preparing  materials  (paper  or  electronic)  for 
distribution  to  the  trainees.  In  most  cases,  the  trainees  determine  the  pace  of  the  training. 
DL  includes  testing  at  specific  points  within  the  instructional  sequence,  with  the  tests 
evaluated  by  the  instructors  who  provide  feedback  and  re-direction  to  the  individual 
trainees  as  needed.  The  trainees  may  maintain  periodic  bi-directional  contact  with  the  in¬ 
structor  via  Internet,  telephone,  video-conference,  etc.,  enabling  the  trainees  to  discuss  is¬ 
sues  and  seek  clarifications  of  the  instructional  materials,  and  enabling  the  instructor  to 
better  assess  the  progress  of  the  trainees. 

DL  can  be  highly  effective  for  all  but  the  most  complex  products.  Preparation  cost  is 
somewhat  higher  than  for  classroom  training  due  to  the  need  to  adapt  training  materials 
for  remote  access  and  use.  Delivery  cost  is  somewhat  higher  than  typical  classroom  train¬ 
ing  due  to  the  infrastructure  support  activities.  On  the  other  hand,  a  single  instructor  can 
typically  interact  with  more  distant  learning  trainees  than  classroom  trainees.  The  elimi¬ 
nation  of  travel  costs  for  the  trainees  also  provides  a  significant  delivery  cost  advantage. 

•  Classroom  Training  -  Unlike  self-training  and  CBT,  classroom  training  is  a  group  activ¬ 
ity  rather  than  an  individual  activity.  It  is  typically  instructor-paced,  although  a  good  in¬ 
structor  will  adapt  the  pace  of  the  training  to  the  abilities  of  the  majority  of  his  students. 
Classroom  training  can  be  designed  using  proven  instructional  design  methods  that  estab¬ 
lish  learning  objectives,  provide  instruction  to  support  those  objectives,  and  verify  the 
trainee’s  attainment  of  those  objectives.  Typically,  classroom  training  provides  instruction 
in  accordance  with  a  defined  curriculum  using  lectures,  discussions,  and  various  presen¬ 
tation  aids  (e.g.  audio,  video).  Hands  on  training  may  be  included  using  either  the  actual 
product,  or  training  aids  (e.g.  simulators,  mock-ups).  The  training  often  includes  exer¬ 
cises  to  provide  guided  practice,  aiding  the  trainee’s  absorption  of  knowledge.  At  specific 
points  within  the  instructional  sequence,  the  instructor  tests  the  knowledge  of  the  trainee, 
and  provides  feedback. 

Classroom  training  can  be  highly  effective  for  all  types  of  products.  Preparation  cost  is 
higher  than  that  for  self-learning,  but  less  than  that  for  CBT  or  distance  learning.  Deliv¬ 
ery  cost  is  significantly  higher  than  self-learning  or  CBT.  While  delivery  cost  may  be  less 
than  distance  learning,  total  cost,  including  the  travel  costs  of  the  students  to  get  to  the 
class  site,  may  be  higher.  In  the  DoD,  classroom  training  typically  has  to  be  coordinate 
with  a  specific  training  unit.  The  training  unit  should  be  considered  a  stakeholder  in  the 
acquisition. 
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•  Field  Training  -  Field  training  is  accomplished  by  sending  a  trained  instructor  to  (or 
near)  the  site  where  the  students  will  use  the  product.  The  trainer  instructs  the  students, 
either  individually,  or  in  a  group,  on  the  use  and/or  maintenance  of  the  product.  Field 
training  is  accomplished  in  accordance  with  a  defined  curriculum.  While  it  may  include 
instruction  using  lectures  and  discussions,  it  is  predominantly  hands-on  training  using  the 
actual  product.  It  is  typically  instructor-paced,  rather  than  student-paced;  although  a  good 
instructor  will  adapt  the  pace  of  the  training  to  the  abilities  of  the  students.  Typically,  stu¬ 
dents  verify  achievement  of  the  learning  objectives  directly  by  demonstration  of  skills  to 
the  instructor. 

Field  training  can  be  highly  effective  for  all  types  of  products  due  to  the  one-on-one  in¬ 
teraction  with  the  instructor  in  the  product’s  intended  environment.  Preparation  cost  is  of¬ 
ten  less  than  classroom  training,  distance  training,  or  CBT.  Delivery  cost  is  significantly 
higher  than  all  of  the  alternatives,  since  it  involves  transporting  the  trainer  to  the  training 
site,  and  training  students  individually  or,  at  most,  in  small  groups. 

The  strategic  choices  and  drivers  of  the  Training  strategy  element  are  summarized  in  the 
strategy  profile  shown  in  Table  12. 
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Table  12:  Training  Strategy  Profile 


Strategy  Element 

Range 

Self-Training 

Computer-Based  Training 

Distance  Learning 

Classroom  Training 

Field  Training 

Principal  driver 

1.  Software  Criticality 

5.  Life  Cycle  Category  Drivers 

influences 

2.  Acquisition  Environment  Cat. 

a.  Product  Definition  and 

(from  Table  6) 

Drivers 

Specification 

a.  Policies  and  Mandates 

i.  Requirements  Volatility 

b.  Supplier  Availability 

ii.  Requirements  Understanding 

3.  Programmatic  Category  Drivers 

iii.  Quality’  Attribute  Definition 

a.  Mission  Needs  and  Scope 

Quality 

b.  Funding 

iv.  Interoperability’ 

i.  Funding  Constraints 

b.  Architecture  and  Design 

ii.  Funding  Profile 

i.  Precedence 

c.  Schedule 

ii.  Quality’  Attribute  Constraints 

i.  Schedule  Constraints 

iii.  Technology’  Readiness 

ii.  Urgency 

iv.  Legacy  Considerations 

4.  Organizational  Category  Drivers 

v.  COTS  /  GOTS  /  Reuse 

a.  Program  Management  Office 

d.  Deployment 

Capabilities 

i.  Number  of  Sites 

i.  PMO  Staff  Skills 

ii.  User  Readiness 

ii.  PMO  Staff  Capacity 

iii.  Maintainer  Readiness 

Hi.  PMO  Staff  Stability 

iv.  Transition  /  Data  Migration 

b.  Stakeholders 

e.  Maintenance  and  Support 

i.  Number  and  Diversity 

i.  Number  of  System 

ii.  Level  of  Engagement  (re¬ 

Configurations 

sponsiveness  and  quality) 

ii.  Update  Readiness 

iii.  Level  of  Agreement 

iii.  Support  Duration 

c.  Supplier  Capability 

iv.  Re-Competition  Readiness 

i.  Supplier  Staff  Skills 

v.  Operational  Environment 

ii.  Supplier  Staff  Capacity’ 

vi.  Legacy  Considerations 

iii.  Supplier  Staff  Stability 

vii.  Complexity’  of  Data  Rights 

iv.  Supplier  Performance  to  Date 

f.  Disposal 
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4.8  Product  Support  Strategy:  Source  of  Support 
Strategy  Element 

Note:  Some  of  the  information  contained  in  this  section  is  familiar  to  most  experienced  ac¬ 
quisition  PMO  staff.  It  is  reiterated  in  this  report  to  lend  coherence  to  the  authors’  discussion 
of  acquisition  strategy  elements. 

After  the  product  is  developed,  tested,  and  delivered,  the  job  of  the  acquisition  PMO  is  fin¬ 
ished,  but  that  is  not  the  end  of  the  program.  Long  after  the  acquisition  PMO  has  been  closed, 
the  product  must  be  maintained.  New  trainers  must  be  trained  to  train  the  new  users  and 
maintainers.  Newly  discovered  product  defects  must  be  corrected.  Product  upgrades  must  be 
developed  and  fielded.  At  the  end  of  its  useful  life,  the  product  must  be  disposed  of.  In  short, 
the  product  must  be  supported  throughout  its  remaining  life  cycle.  While  this  activity  often 
becomes  the  responsibility  of  an  Operations  PMO,  it  must  first  be  addressed  by  the  acquisi¬ 
tion  PMO.  Failure  of  the  acquisition  PMO  to  adequately  address  deployment,  support,  and 
disposal  of  the  product  may  make  these  tasks  unreasonably  difficult  and  costly,  or  even  im¬ 
possible,  to  perform. 

Answering  the  question  “Who  will  support  this  product?”  is  a  key  factor  in  the  development 
of  a  support  strategy.  Sources  of  product  support  range  from  various  forms  of  organic  support 
(support  provided  from  within  the  government)  to  support  provided  solely  by  a  contractor,  as 
shown  in  Figure  10  [OUSD  AT&L  01], 


More  organic 


More  cofTfrercia 


r  ^ 

Organic^ 

r  ^ 

Contractor 

V  J 

Organic 
providers 
responsible 
for  majority  of 
support 

Public-private 

partnering 

opportunities 

Contractor 
responsible 
for  majority 
of  support 

PBL  strategies  driven  by  MOUs  with  the  warfighters 
will  vary  along  this  spectrum  depending  on 
■  Age  of  system  (phase  in  life  cycle) 

•  Existing  support  infrastructure 

•  Organic  and  commercial  capabilities 

•  Legislative  and  regulatory  constraints 


Examples  of  partnering  agreements: 

-  Total  system  performance  responsibility 

•  Govemmentlindustry  partnenng 

•  Service  level  agreements 

•  Performance-based  agile  logistics  support 

•  Prrre  vendor  support 

•  Contractor  delivery  system 


Figure  10:  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics’  Depiction  of  the  Spectrum  of  Performance  Based  Logistics 
(PBL)  Strategies 
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Candidate  support  strategies  include 

•  Contractor  Logistics  Support  (CLS)  -  For  CLS,  product  support  is  contracted  to  a  com¬ 
mercial  contractor.  This  contractor  may  be  the  original  developer  of  the  product,  an  or¬ 
ganization  that  specializes  in  logistics  support,  etc.  In  fact,  the  support  activities  may 
even  be  divided  among  multiple  contractors;  one  for  field  maintenance,  another  for  up¬ 
grade  development,  and  yet  another  for  disposal.  The  role  of  the  contractors  is  defined  in 
the  support  contract  from  the  Operations  PMO.  It  includes  a  definition  of  the  scope  of  the 
support  effort,  as  well  as  requirements  for  the  quality  of  support  (response  time,  failure 
rates,  mean  time  to  repair,  etc.).  Cost-plus-incentive-fee  or  cost-plus-award-fee  contracts 
are  commonly  used  for  CLS.  Fixed-price  contracts  are  unusual  due  to  the  uncertainty  of 
the  amount  of  effort  that  is  required. 

A  CLS  strategy  places  demands  on  the  acquirer.  The  acquirer  must  ensure  sufficient 
availability  and  data  rights  to  technical  information  from  the  developer  to  enable  the  sup¬ 
port  contractor  to  perform  the  maintenance  and  upgrade  tasks.  The  acquirer  must  ensure 
that  the  reliability  and  maintainability  of  the  product  are  adequate  to  enable  the  support 
contractor  to  fulfill  his  mission.  The  acquirer  must  also  ensure  that  sufficient  training  is 
available. 

CLS  can  be  a  viable  support  option  for  nearly  all  products.  It  enables  the  PMO  to  draw 
upon  the  technical  and  management  strengths  of  the  defense  industry  to  support  fielded 
products.  In  cases  where  the  acquirer  has  not  negotiated  with  the  developer  for  sufficient 
data  rights  and  access  to  technical  information,  CLS  with  the  developer  as  the  support 
contractor  may  be  the  ONLY  viable  strategy.  CLS  can  be  an  expensive  support  option; 
however,  with  appropriate  incentives,  the  cost  can  be  managed. 

•  Organic  Support  -  Organic  support  may  take  several  forms,  depending  upon  which  or¬ 
ganization  within  the  government  provides  the  support,  as  shown  below: 

-  PMO  Support  -  Product  support  can  be  provided  directly  from  the  Operations  PMO 
by  PMO  staff  trained  in  the  support  of  the  product.  Like  CLS,  a  PMO  support  strat¬ 
egy  also  places  demands  upon  the  acquirer.  The  acquirer  must  again  ensure  sufficient 
availability  and  data  rights  to  technical  information  from  the  developer  to  enable  the 
PMO  support  staff  to  perform  maintenance  and  upgrade  tasks.  The  acquirer  must  en¬ 
sure  that  the  reliability  and  maintainability  of  the  product  are  adequate  to  enable  the 
PMO  support  staff  to  fulfill  their  mission.  The  acquirer  must  also  ensure  that  suffi¬ 
cient  training  is  available. 

Although  PMO  support  is  not  used  frequently  due  to  chronic  understaffing  of  PMOs, 
it  can  be  a  viable  support  option  for  products  of  low  to  medium  complexity.  PMO 
support  can  be  a  relatively  inexpensive  support  option;  however,  the  quality  of  sup¬ 
port  is  highly  dependent  upon  the  skill  and  training  of  the  PMO  support  staff. 

-  Indigenous  Support  -  Some  products  can  be  supported  primarily  by  the  units  to 
which  they  are  fielded.  For  example,  transportation  vehicles  are  often  maintained  by 
an  on-base  maintenance  facility  (i.e.,  motor  pool)  staffed  by  base  staff.  Like  CLS,  an 
indigenous  support  strategy  places  the  same  demands  upon  the  acquirer.  The  acquir¬ 
ing  PMO  must  ensure  sufficient  availability  and  data  rights  to  technical  information 
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from  the  developer,  that  the  reliability  and  maintainability  of  the  product  are  ade¬ 
quate,  and  that  sufficient  training  is  available. 

Indigenous  support  works  effectively  on  relatively  simple  products  that  can  be  sup¬ 
ported  without  extensive  training,  and  it  can  be  an  inexpensive  option. 

-  Depot  Support  -  Product  support  can  be  provided  by  depots  or  other  DoD  support  fa¬ 
cilities. 

Like  CLS,  a  depot  support  strategy  again  places  the  same  demands  upon  the  acquirer. 
The  acquiring  PMO  must  ensure  sufficient  availability  and  data  rights  to  technical  in¬ 
formation  from  the  developer,  that  the  reliability  and  maintainability  of  the  product 
are  adequate,  and  that  sufficient  training  is  available. 

Depot  support  works  effectively  on  products  widely  used  throughout  the  DoD.  It  is 
applicable  to  products  at  all  levels  of  complexity.  With  sufficient  volume  of  products, 
the  depot  can  afford  the  investment  needed  to  establish  and  maintain  a  support  capa¬ 
bility.  Additionally,  the  depot  can  establish  agreements  with  the  product  developer  to 
address  recurring  failures  and  product  upgrades. 

Many  combinations  of  organic  and  contractor  support  are  possible. 

The  strategic  choices  and  drivers  of  the  Source  of  Support  strategy  element  are  summarized 
in  the  strategy  profile  shown  in  Table  13. 
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Table  13:  Source  of  Support  Strategy  Profile 


Strategy 

Element  Range 

Contractor  Logistics  Support 

PMO  Support 

Depot  Support 

Indigenous  Support 

Principal  driver 

1. 

Software  Criticality 

5.  Life  Cycle  Category  Drivers 

influences 

2. 

Acquisition  Environment  Cat.  Drivers 

a. 

Product  Definition  and 

(from  Table  6) 

a.  Policies  and  Mandates 

Specification 

b.  Supplier  Availability 

iv.  Interoperability 

3. 

Programmatic  Category  Drivers 

b. 

Architecture  and  Design 

a.  Mission  Needs  and  Scope 

i.  Precedence 

b.  Funding 

iii.  Technology’  Readiness 

i.  Funding  Constraints 

iv.  Legacy  Considerations 

ii.  Funding  Profile 

v.  COTS  /  GOTS  /  Reuse 

c.  Schedule 

d. 

Deployment 

i.  Schedule  Constraints 

i.  Number  of  Sites 

ii.  Urgency 

ii.  User  Readiness 

4. 

Organizational  Category  Drivers 

iii.  Maintainer  Readiness 

a.  Program  Management  Office 

iv.  Transition  /  Data  Migration 

Capabilities 

e. 

Maintenance  and  Support 

i.  PMO  Staff  Skills 

i.  Number  of  System 

ii.  PMO  Staff  Capacity 

Configurations 

Hi.  PMO  Staff  Stability 

ii.  Update  Readiness 

b.  Stakeholders 

iii.  Support  Duration 

i.  Number  and  Diversity 

iv.  Re-Competition  Readiness 

ii.  Level  of  Engagement 

v.  Operational  Environment 

(responsiveness  and  quality) 

vi.  Legacy  Considerations 

iii.  Level  of  Agreement 

vii.  Complexity  of  Data  Rights 

c.  Supplier  Capability 

f. 

Disposal 

i.  Supplier  Staff  Skills 

ii.  Supplier  Staff  Capacity 

iii.  Supplier  Staff  Stability 

iv.  Supplier  Performance  to  Date 
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5  Formulating  an  Acquisition  Strategy 


5.1  Overview 

In  Section  3,  we  defined  a  set  of  drivers  that  influence  a  program’s  acquisition  strategy  and  in 
Section  4,  we  began  to  codify  an  acquisition  strategy  in  terms  of  strategy  elements  and  corre¬ 
sponding  strategic  choices. 

This  section  uses  the  strategy  profiles  created  for  the  strategy  elements  and  sub-elements 
(Acquisition  Approach,  Business  Considerations,  and  Product  Support)  discussed  in  Section 
4  to  illustrate  a  strategy  development  method.  Note  that  additional  strategy  profiles  and  strat¬ 
egy  elements  may  be  addressed  in  future  research. 

Many  different  drivers  influence  multiple  strategy  elements  and  vice  versa.  Due  to  the  num¬ 
ber  and  the  complexity  of  relationships  between  the  drivers  and  the  strategy  elements,  it  is 
difficult  to  determine  which  strategy  choices  best  fit  the  drivers.  For  a  strategy  that  has  “M” 
strategy  drivers  and  “N”  strategy  elements,  the  number  of  relationships  could  approach  (M  x 
N).  For  example,  if  there  were  10  drivers  and  5  strategy  elements  the  number  of  relationships 
could  approach  50. 

Programs  need  better  ways  to  reason  about  software  risks,  formulate  acquisition  strategies  to 
mitigate  software  risk,  and  evaluate  their  current  acquisition  strategy’s  ability  to  mitigate 
software  risk  in  an  ongoing,  systematic  manner.  In  this  section,  we  provide  a  method  for  per¬ 
forming  a  comparative  analysis  of  the  drivers,  the  strategy  elements  and  their  corresponding 
strategic  choices. 

The  method  we  propose  is  a  graphical,  bi-directional  method  of  examining  and  analyzing  the 
relationships  between  the  drivers  and  the  strategy  element  strategic  choices.  The  method  uses 
a  graphical  element,  slider  bars,  to  support  a  more  systematic  approach  to  reasoning  about 
software  risk.  The  method  is  bi-directional  in  that  it  enables  a  program  to 

1 .  Analyze  strategy  drivers  and  then  choose  strategies  (strategic  choices)  for  each  strategy 
element  to  minimize  the  risks  induced  by  those  drivers. 

2.  Choose  a  strategy  (strategic  choice)  for  each  strategy  element  and  then  analyze  the  risks 
induced  by  the  program’s  drivers. 

In  this  section,  we  explain 

•  the  strategy  development  method  (Section  5.2) 

•  the  concept  of  a  slider  bar  (Section  5.3) 
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•  how  to  diagram  a  program’s  software  acquisition  characteristics,  its  strategy  drivers,  its 
acquisition  strategy  elements  and  corresponding  strategic  choices,  and  the  relationship 
between  the  drivers  and  strategic  choices  (Section  5.4) 

By  evaluating  strategy  drivers  and  acquisition  strategy  elements’  strategic  choices  using  the 
slider  bar  technique,  program  managers  and  others  can  rate  their  program  and  system  to  build 
an  acquisition  profile.  As  a  result,  implementations  will  be  more  sharply  focused  on  mitigat¬ 
ing  underlying  software  risks  and  will  exhibit  higher  levels  of  acceptance,  commitment,  and 
understanding  among  those  who  implement  and  oversee  the  program’s  acquisition  strategy. 

5.2  Strategy  Development  Method 

As  discussed  in  Section  2,  factors  both  within  the  program  and  external  to  the  program  gen¬ 
erate  program  risks.  We  call  these  factors  drivers  and  have  discussed  them  at  length  in  Sec¬ 
tion  3.  The  goal  of  an  effective  acquisition  strategy  is  to  addresses  the  program  risks  gener¬ 
ated  by  these  drivers.  No  strategy  can  be  sufficiently  comprehensive  and  suitably  complex  to 
address  all  of  the  risks  of  a  program.  However,  we  can  formulate  a  strategy  that  addresses  the 
most  significant  risks  and  leave  the  remaining  risks  to  be  addressed  by  the  risk  management 
process  within  the  program,  as  shown  in  Figure  11. 


Figure  11:  Risk  Management  Via  Acquisition  Strategy 

To  use  the  acquisition  strategy  as  a  means  of  mitigating  program  risks,  we  must 

1 .  identify  the  program  risks 

2.  identify  the  elements  of  the  strategy 

3.  understand  the  relationship  between  the  risks  and  the  strategy  elements’  strategic  choices 
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The  presumptions  of  this  acquisition  strategy  development  method  are  that 

•  The  strategy  drivers  can  be  related  to  the  production  of  risk  for  the  program. 

•  The  strategic  choices  within  strategy  elements  can  be  ranked  in  order  of  their  ability  to 
address  the  program’s  identified  classes  of  risk. 

•  The  strategic  choices  within  strategy  elements  can  mitigate  risks  generated  by  specific 
strategy  drivers. 

The  next  three  sections  discuss  these  presumptions  in  more  detail. 

5.2.1  Strategy  Drivers  and  Risk 

Let’s  consider  the  first  premise — strategy  drivers  can  be  related  to  the  production  of  risk  for 
the  program.  For  example,  “software  criticality”  is  one  of  the  drivers  we  defined  and  dis¬ 
cussed  in  Section  3.  Clearly,  the  more  critical  software  is  to  achieve  the  required  program 
performance,  the  more  risk  it  poses  to  the  program.  “Requirements  understanding,”  another 
driver  discussed  in  Section  3,  can  be  considered  in  the  same  manner;  requirements  that  are 
well  understood  pose  less  risk  to  a  program  than  requirements  that  are  not  understood. 

5.2.2  Strategy  Elements  and  Risk 

Let’s  consider  the  second  premise — strategy  choices  within  strategy  elements  can  be  ranked 
in  order  of  their  ability  to  address  the  program’s  identified  classes  of  risk.  For  the  Acquisition 
Approach  strategy  element,  three  strategy  choices  are  suggested  in  Section  4:  single-step, 
evolutionary/incremental,  and  evolutionary/spiral.  The  question  that  we  need  to  address  is 
“Flow  do  these  three  choices  compare  in  terms  of  mitigating  program  risk?”  To  answer  this 
question,  one  line  of  reasoning  is  as  follows: 

•  The  evolutionary/spiral  acquisition  approach  was  specifically  developed  as  a  means  of 
addressing  program  risk.  It  enables  the  acquirer  to  perform  a  risk-driven  partitioning  of 
the  program  into  sequential  acquisition  spirals.  Each  spiral  is  defined  to  address  the  most 
significant  risk  issues  of  the  program  at  that  phase  of  its  development.  In  this  manner,  the 
acquirer  can  address  and  perform  early  mitigation  of  the  highest  risks  to  the  program. 

This  acquisition  approach  is  particularly  good  at  dealing  with  incomplete  and  poorly  de¬ 
fined  requirements  in  that  early  spirals  can  provide  useful  demonstrations  of  capabilities 
to  the  stakeholders,  enabling  them  to  better  understand  the  direction  of  the  development, 
and  clarify  their  statements  of  need. 

•  The  evolutionary/incremental  acquisition  approach  is  also  a  means  of  addressing  program 
risk.  It  enables  deferment  of  some  portions  of  the  development  to  later  times  in  the  pro¬ 
gram  to  permit  time  for  technology  development,  staff  development,  and  so  on.  It  is  not 
particularly  suitable  for  dealing  with  incomplete  and  poorly  defined  requirements  in  that 
all  planned  increments  are  based  upon  needs  defined  at  the  start  of  the  program. 

•  The  single-step  acquisition  approach  is  least  capable  of  addressing  program  risk  largely 
because  it  assumes  a  low-risk  program  to  begin  with.  It  requires  the  development  of  a 
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start-to-fmish  execution  plan  based  on  needs  defined  at  the  start  of  the  program.  As  such, 
it  is  least  suitable  for  dealing  with  incomplete  and  poorly  defined  requirements. 

In  light  of  this  discussion,  we  could  rank  the  strategy  choices  as  follows: 

Acquisition  Approach  Strategy  Risk  Mitigation  Capabilities 

Single-Step  Low 

Evolutionary/Incremental  Medium 

Evolutionary/Spiral  High 


As  a  second  example,  let  us  look  at  the  Business  Considerations:  Competition  strategy  ele¬ 
ment.  Section  4  provides  three  candidate  strategy  choices:  Full  and  Open  Competition,  Full 
and  Open  Competition  After  Exclusion  of  Sources,  and  Sole  Source  Contracting. 

We  could  rank  the  risk  mitigation  capabilities  of  these  strategy  choices,  according  to  the  fol¬ 
lowing  reasoning: 

•  A  Full  and  Open  Competition  strategy  provides  the  greatest  degree  of  mitigation  for  risks 
to  achieving  desired  performance.  It  mitigates  performance  risk  by  gathering  technical 
and  management  approaches  from  the  widest  population  of  bidders.  It  mitigates  cost  and 
schedule  risk  by  placing  all  bidders  in  a  competitive  environment  encouraging  them  to 
plan  and  execute  with  optimum  efficiency. 

•  A  Full  and  Open  Competition  After  Exclusion  of  Sources  strategy  is  less  robust  at  miti¬ 
gating  program  risk.  Capable  sources  are  excluded  based  upon  goals  for  Small  Business 
utilization,  goals  for  disadvantaged  business  utilization,  etc.  Because  the  population  of 
respondents  is  reduced,  mitigation  of  performance  risk  is  also  reduced.  Because  the  com¬ 
petitive  environment  is  limited,  cost  and  schedule  risk  mitigation  is  likewise  limited. 

•  A  Sole  Source  Contracting  strategy  offers  the  least  degree  of  risk  mitigation.  Only  one 
technical  approach  from  one  supplier  is  considered.  No  competition  is  present  to  encour¬ 
age  efficient  operation  and  mitigate  cost  and  schedule  risk. 

In  light  of  this  discussion,  we  could  rank  the  strategy  choices  as  follows: 

Competition  Strategy  Risk  Mitigation  Capabilities 

Sole  Source  Contracting  Fow 

Full  and  Open  Competition  after  Medium 

Exclusion  of  Sources 

Full  and  Open  Competition  High 


The  examples  above  are  of  a  notional  nature.  Other  circumstances  can  be  envisioned  that 
would  lead  to  different  orderings  when  ranking  the  strategy  choices.  As  such,  the  ranking 
process  is  highly  contextual  and  requires  careful  consideration  of  the  circumstances  of  each 
project. 
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5.2.3  Relating  Strategy  Drivers  to  Strategy  Elements 

From  the  preceding  discussion,  it’s  clear  that  strategy  drivers  influence  program  risk.  We  can 
also  observe  how  some  strategic  choices  are  better  able  to  address  risk  than  others.  Relating 
both  strategy  drivers  and  strategy  elements  to  risk  provides  a  means  for  correlating  drivers 
and  strategic  choices;  programs  with  higher  risks,  as  evidenced  by  the  strategy  drivers, 
should  employ  strategic  choices  better  able  to  address  program  risks.  Flowever,  to  accomplish 
this,  we  must  first  know  which  risks  can  be  mitigated  by  strategy  element  strategic  choices. 

As  noted  in  Section  5.2.2,  these  linkages  will  differ  for  different  programs.  As  such,  the  link¬ 
ages  must  be  carefully  evaluated  in  the  context  of  each  individual  program.  Although  a  tedi¬ 
ous  process,  we  used  a  group  brainstorming  method  to  understand  these  relationships.  Ad¬ 
dressing  each  strategy  element  individually,  we  first  reviewed  the  range  of  strategy  choices 
determined  for  that  element.  Subsequently,  we  discussed  the  impact  of  each  individual  strat¬ 
egy  driver  upon  the  strategy  element.  Much  of  the  discussion  consisted  of  postulating  pro¬ 
gram  scenarios  to  expose  the  relationships  between  the  driver  and  the  strategy  element. 

Table  14  and  Table  15  provide  a  notional  summary  of  this  relationship  for  a  software¬ 
intensive  system.  The  linkages  between  the  strategy  drivers  and  strategy  elements  are  charac¬ 
terized  notionally  as 

•  Strong  (S)  -  A  significant  correlation  exists  between  the  risk  posed  by  the  strategy  driver 
and  the  risk  mitigation  capabilities  of  the  strategy  element. 

•  Medium  (M)  -  A  moderate  correlation  exists  between  the  risk  posed  by  the  strategy 
driver  and  the  risk  mitigation  capabilities  of  the  strategy  element. 

•  Weak  or  None  (no  entry)  -  A  small  correlation  or  no  correlation  exists  between  the  risk 
posed  by  the  strategy  driver  and  the  risk  mitigation  capabilities  of  the  strategy  element. 
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Table  14:  Strategy  Driver  -  Strategy  Element  Mapping 

STRATEGY  ELEMENTS 


Pgrm 

Struc¬ 

ture 

Acquisition  Approach 

Business 

Considerations 

Risk  Management 

Information  Assurance-Development  Source 

Test  and  Evaluation 

Produ 

Suppo 

Strate 

ct 

rt 

?y 

KEY 

M 

S 

Weak  or  No  Linkage 

Medium  Linkage 

Strong  Linkage 

Milestone  Decision  Points 

Acquisition  Phases 

Competition 

Solicitation 

Source  Selection 

Contract  Approach 

Training 

Installation 

Source  of  Support 

STRATEGY  DRIVERS 

Software  Criticality 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

M 

Acquisition  Environment 

Policies  and  Mandates 

S 

s 

S 

S 

S 

S 

S 

S 

s 

S 

S 

S 

Supplier  Availability 

S 

s 

s 

S 

S 

S 

S 

s 

M 

M 

S 

Programmatic  Category  Drivers 

Mission  Needs  and  Scope 

s 

s 

s 

S 

S 

s 

s 

M 

s 

s 

S 

S 

s 

Funding 

Funding  Constraints 

s 

s 

s 

S 

S 

s 

s 

M 

s 

s 

S 

S 

s 

Funding  Profile 

s 

s 

s 

S 

M 

s 

M 

M 

M 

M 

s 

Schedule 

Schedule  Constraints 

s 

s 

s 

S 

s 

s 

s 

M 

M 

s 

S 

S 

s 

Urgency 

s 

s 

s 

S 

s 

s 

s 

M 

M 

s 

S 

S 

s 

Organizational  Category  Drivers 

Program  Management  Office 
Capabilities 

PMO  Staff  Skills 

M 

M 

M 

M 

M 

s 

M 

S 

M 

M 

M 

M 

s 

PMO  Staff  Capacity 

M 

M 

M 

M 

M 

s 

M 

S 

M 

M 

M 

M 

s 

PMO  Staff  Stability’ 

M 

M 

M 

M 

M 

s 

M 

s 

M 

M 

M 

M 

s 

PMO  Process  Focus 

s 

s 

S 

Stakeholders 

Number  and  Diversity 

S 

s 

s 

S 

M 

M 

S 

S 

s 

Level  of  Engagement 

s 

s 

s 

s 

M 

M 

S 

S 

s 

Level  of  Agreement 

s 

s 

s 

s 

M 

M 

s 

s 

s 

Supplier  Capability 

Supplier  Staff  Skills 

s 

s 

s 

S 

S 

s 

s 

s 

Supplier  Staff  Capacity 

s 

s 

s 

s 

S 

s 

s 

s 

Supplier  Staff  Stability 

s 

s 

s 

s 

s 

s 

s 

s 

Supplier  Performance  to  Date 

s 

s 

s 

s 

s 

s 

s 

s 
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Table  15:  Strategy  Driver  -  Strategy  Element  Mapping  (continued) 


STRATEGY  ELEMENTS 

Prog¬ 

ram 

Struc¬ 

ture 

Acquisition  Approach 

Business 

Considerations 

Risk  Management 

Information  Assurance  -  Dev’t  Source 

Test  and  Evaluation 

Product 

Support 

Strategy 

KEY 

M 

S 

Weak  or  No  Linkage 

Medium  Linkage 

Strong  Linkage 

Milestone  Decision  Points 

Acquisition  Phases 

Competition 

Solicitation 

Source  Selection 

Contract  Approach 

Training 

Installation 

Source  of  Support 

STRATEGY  DRIVERS 

Life-Cycle  Category  Drivers 

Product  Definition  and  Specifi¬ 
cation 

Requirements  Volatility 

S 

S 

S 

S 

S 

S 

S 

S 

S 

M 

M 

M 

Requirements  Understanding 

S 

S 

S 

S 

S 

S 

S 

s 

S 

M 

M 

M 

Quality  Attribute  Definition 
Quality 

s 

s 

s 

S 

s 

s 

M 

s 

S 

M 

Interoperability 

s 

s 

s 

S 

s 

s 

s 

S 

s 

S 

S 

S 

S 

Architecture  and  Design 

Precedence 

s 

s 

s 

S 

s 

s 

s 

s 

s 

s 

s 

S 

S 

Quality  Attribute  Constraints 

s 

s 

s 

S 

s 

s 

s 

M 

s 

s 

M 

Technology’  Readiness 

s 

s 

s 

S 

s 

s 

s 

S 

s 

s 

S 

s 

S 

Legacy  Considerations 

s 

s 

s 

S 

s 

s 

s 

s 

s 

s 

s 

s 

S 

COTS  /  GOTS  /  Reuse 

s 

s 

s 

S 

s 

s 

s 

s 

s 

s 

s 

s 

S 

Verification  and  Test 

Test  Environment  Complexity 

M 

M 

s 

Test  Environment  Availability 

M 

M 

s 

Number  of  System 
Configurations 

M 

M 

s 

Deployment 

Number  of  Sites 

M 

M 

s 

s 

s 

S 

User  Readiness 

M 

M 

s 

s 

s 

S 

Maintainer  Readiness 

M 

M 

s 

s 

s 

S 

Transition  /  Data  Migration 

M 

M 

s 

s 

s 

S 

Maintenance  and  Support 

Number  of  System 
Configurations 

M 

s 

s 

s 

S 

Update  Frequency 

M 

s 

s 

s 

S 

Support  Duration 

M 

s 

s 

s 

S 

Re-Competition  Readiness 

M 

M 

S 

Operational  Environment 

M 

s 

s 

s 

S 

Legacy  Considerations 

M 

M 

M 

s 

S 

Complexity  of  Data  Rights 

M 

S 

s 

S 

Disposal 

M 

S 

s 

M 
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5.3  Slider  Bar  Technique 

Slider  bars  are  a  graphical  way  to  visualize  a  range  or  continuum  of  values.  The  conceptual 
elements  of  a  slider  bar  are 

•  a  label  defining  the  domain  of  the  slider  bar 

•  endpoints  defining  the  range  of  the  slider  bar 

•  a  “slider”  that  indicates  the  current  value  within  the  range  defined  by  the  endpoints 

For  example,  if  we  were  to  represent  the  speaker  volume  of  a  car  stereo,  the  endpoints  might 
be  “soft”  and  “loud,”  as  shown  in  Figure  12.  The  “slider”  shows  you  where  the  current  value 
of  the  volume  is  on  the  continuum  (about  half-volume). 


Volume 


◄ - 

Soft 

- ► 

Loud 

Figure  12:  Stereo  Slider  Bar  Example 

5.4  Using  Slider  Bars  to  Profile  Strategy  Drivers, 
Acquisition  Strategies,  and  Their  Relationships 

Slider  bars  can  be  used  to  visually  represent  program  strategy  drivers  and  many  acquisition 
strategies.  They  offer  the  ability  to  relate  strategy  drivers  to  strategy  choices,  using  program 
risk  as  a  common  denominator.  Slider  bars  can  be  used  to  graphically  visualize  a  program’s 
software  acquisition  profile  and  frame  a  discussion  about  a  program’s  strategy  drivers,  risks, 
and  acquisition  strategy.  After  the  program’s  strategy  drivers  have  been  profiled,  the  slider 
bars  can  be  used  to  reason  about  the  efficacy  of  a  given  acquisition  approach  and  its  ability  to 
address  the  program’s  software  risks. 

This  section  introduces  readers  to  the  process  step-by-step.  The  first  two  sections  provide 
some  of  the  basic  steps,  which  build  up  to  more  complex  uses  that  are  explained  in  the  last 
two  sections. 

•  Section  5.4.1  explains  how  to  use  slider  bars  to  represent  a  program’s  software  drivers. 

•  Section  5.4.2  explains  how  to  use  slider  bars  to  represent  acquisition  strategies  that  have 
a  range  of  values. 

•  Section  5.4.3  explains  how  to  evaluate  strategy  drivers  to  help  formulate  an  acquisition 
strategy.  It  offers  a  seven-step  approach  to  help  guide  acquisition  strategy  development. 

•  Section  5.4.4  discusses  an  approach  to  help  evaluate  an  existing  strategy  given  the  pro¬ 
gram’s  current  software  acquisition  profile. 
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Programs  need  better  ways  to  reason  about  software  risks,  formulate  acquisition  strategies  to 
mitigate  software  risk,  and  evaluate  the  ability  of  current  acquisition  strategies  to  mitigate 
software  risk  in  an  ongoing,  disciplined  manner.  By  using  the  slider  bar  technique  to  repre¬ 
sent  its  software  acquisition  profile,  a  program  can  gain  valuable  insight  into  its  software  risk. 

5.4.1  Diagramming  Strategy  Drivers 

Section  3  described  the  software  risk  categories  and  drivers  in  detail.  This  section  explains 
how  to  diagram  strategy  drivers  using  slider  bars.  Each  driver  can  be  represented  as  a  contin¬ 
uum  on  a  slider  bar.  Each  slider  bar  consists  of  two  extremes  of  behavior  that  characterize  the 
driver. 

Figure  1 3  displays  a  template  for  visually  representing  a  driver  using  a  continuum  bar  with 
appropriate  annotations. 


r 


STRATEGY  DRIVER 

RANGE 

Drivei  Name 

s 

Endpointl 

End  |>oint2 

Driver  Nome 


HinCREASING  RISKh 


End|)oint2 

J 


Figure  13:  Slider  Bar  Template 

The  “Driver  Name”  is  the  name  of  the  program  driver  being  evaluated.  The  endpoints  deline¬ 
ate  the  continuum  for  the  driver.  For  example,  if  the  driver  is  “Software  Criticality”  the  end¬ 
points  could  be  “Very  Low”  and  “Very  FTigh,”  as  shown  in  Figure  14. 

[increasing  risk!  ► 

Very  Low  |  |  ^  |  I  I  I  I  Very  High 


STRATEGY  DRIVER 

RANGE 

Software  Criticality 

Very  Low 

Very  High 

Figure  14:  Software  Criticality  Slider  Bar 

Next,  the  program  manager  or  other  program  personnel  identify  and  mark  a  point  on  each 
slider  that  best  represents  the  program’s  software  risk  exposure  for  that  element.  The  person 
or  people  making  the  selection  should  be  able  to  defend  why  the  particular  location  on  the 
slider  bar  was  selected.  The  selected  point  on  a  slider  bar  denotes  the  assessment  of  the  driver 
by  an  individual  or  a  group.  This  technique  is  subjective.  It  is  not  meant  to  provide  a  defini¬ 
tive  and  quantified  assessment  of  program  risk;  rather,  it  is  meant  to  be  used  as  a  tool  to  spur 
discussion  and  to  clarify  thought.  For  example,  a  program  manager  might  assess  her  program 
at  a  different  point  on  the  slider  bar  than  her  manager  or  direct-reports.  This  could  result  in  a 
discussion  about  what  the  true  status  or  risk  is  to  the  program  regarding  that  driver,  leading  to 
a  better  understanding  of  the  risk. 


As  a  further  example  of  using  slider  bars,  we  will  consider  three  acquisition  projects: 
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1.  Project  1  is  the  acquisition  of  a  replacement  Jeep.  The  product  contains  very  little  soft¬ 
ware.  The  requirements  are  well  known  and  major  changes  are  not  expected. 

2.  Project  2  is  the  acquisition  of  a  new  armored  vehicle.  Similar,  but  not  identical,  vehicles 
have  been  acquired  previously.  The  product  contains  a  moderate  amount  of  software  and 
a  few  new  software  requirements,  which  are  not  fully  defined,  are  expected. 

3.  Project  3  is  the  acquisition  of  a  new  unmanned  autonomous  ground  vehicle  (UAGV).  No 
vehicles  with  the  required  capabilities  exist  and  the  technology  needed  to  produce  the 
vehicle  is  not  fully  developed.  The  vehicle  requires  a  moderate  amount  of  software.  The 
system  requirements  are  not  well  defined;  therefore  the  software  requirements  are  not 
defined  (or  can’t  be  defined). 

Figure  15  shows  how  our  assessment  of  requirements  volatility  for  these  three  projects  early 
in  the  acquisition  might  be  marked.  As  time  progresses  and  the  new  software  requirements 
become  better  defined,  the  assessment  could  move  to  the  left  on  the  slider  bar  during  the  it¬ 
erative  process  of  profiling  the  program  and  evaluating  the  acquisition  strategy. 


Requirements  Volatility 

Volatile 

Stable 

Requirements  Volatility 

Volatile 

Stable 

Requirements  Volatility 

Volatile 

Stable 

Volatile 

Volatile 

Volatile 


Figure  15:  Vehicle  Acquisition  Example 

Appendix  A  provides  a  hard-copy  template  for  profiling  the  strategy  drivers  discussed  in  Sec¬ 
tion  3.  In  addition,  the  ASDT  is  available  for  download  on  the  SEI  Publications  Web  site  at 
http://www.sei.cmu.edu/publications/publications.html.  The  ASDT  is  a  Microsoft  Excel- 
based  workbook  that  supports  a  more  automated  approach  for  profiling  strategy  drivers. 

5.4.2  Diagramming  Acquisition  Strategy  Elements 

Slider  bars  can  also  be  used  to  graphically  represent  different  strategic  choices  for  a  specific 
strategy  element.  Slider  bars  cannot  be  used  to  diagram  all  strategy  elements,  but  they  can  be 
used  to  visualize  strategies  that  have  a  risk  mitigation  gradation  associated  with  them.  For 
many  acquisition  strategy  elements,  the  strategic  choices  can  be  ranked  based  upon  their  risk 
mitigation  capabilities,  as  discussed  in  Section  5.2.2.  These  strategy  elements  can  be  repre¬ 
sented  by  slider  bars  in  a  manner  similar  to  that  of  the  strategy  drivers 

Section  4  described  the  acquisition  strategy  elements  at  a  high  level.  This  section  explains 
how  to  diagram  acquisition  strategic  choices  for  a  particular  strategy  element  using  slider 
bars. 
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As  examples,  consider  the  Acquisition  Approach  and  Business  Considerations:  Competition 
strategy  elements  and  their  associated  risk  mitigation  capabilities,  as  discussed  in  Section 
5.2.2: 


Acquisition  Approach  Strategy 

Single-Step 

Evolutionary/Incremental 

Evolutionary/Spiral 


Risk  Mitigation  Capabilities 

Low 

Medium 

High 


Competition  Strategy 

Sole  Source  Contracting 

Full  and  Open  Competition  after  Exclusion  of 
Sources 

Full  and  Open  Competition 


Risk  Mitigation  Capabilities 

Low 

Medium 

High 


These  strategy  element  strategic  choices  can  be  represented  by  slider  bars,  as  shown  in  Figure 
16  and  Figure  17. 

I  STRATEGY  ELEMENT:  [Acquisition  Approach  I 


STRATEGIES 

1 

Evolutionary/Spiral 

2 

3 

4 

Evolutionarydncremental 

5 

6 

7 

Single-Step 

8 

1 


Strategy  Slider  Bar 

INCREASING  RISK 

MITIGATION  CAPABILITY 

Figure  16:  Slider  Bar  for  Acquisition  Approach  Strategy  Element 
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I  STRATEGY  ELEMENT:  ICompetition  I 


STRATEGIES 

1 

Full  and  Open  Competition 

2 

3 

4 

Full  and  Open  Competition  After  Exclusion  of  Sources 

5 

6 

7 

Sole  Source  Contracting 

8 

Strategy  Slider  Bar 

INCREASING  RISK 

MITIGATION  CAPABILITY 

Figure  1 7;  Slider  Bar  for  Competition  Strategy  Element 

5.4.3  Developing  a  Strategy  by  Evaluating  Strategy  Drivers 

By  relating  strategy  drivers,  strategy  elements,  and  corresponding  strategic  choices  to  risk,  as 
illustrated  in  the  previous  two  sections,  we  have  established  a  basis  for  correlating  them. 
When  developing  an  acquisition  strategy,  the  process  of  mapping  strategy  drivers  to  strategy 
element  strategic  choices  consists  of  nine  steps: 

1.  Define  the  objectives  of  the  acquisition. 

2.  Identify  and  evaluate  the  factors  that  drive  the  program. 

3.  Decompose  the  strategy  into  individual  strategy  elements. 

4.  For  the  selected  strategy  element,  identify  the  potential  strategic  choices,  and  rank  them 
in  order  of  their  risk  mitigation  capabilities  for  your  particular  program. 

5.  Evaluate  the  strategy  drivers  for  the  program  to  identify  those  that  influence  the  strategy 
element  through  the  introduction  of  risk  that  may  be  mitigated  by  strategy  element. 

6.  Define  the  relationship  between  the  risk  generated  by  the  strategy  driver  and  the  risk 
mitigation  capabilities  of  the  strategy  element. 

7.  Map  the  driver  evaluations  from  Step  1  to  the  strategy  element  using  the  slider  bars. 

8.  Choose  the  strategy  that  best  mitigates  the  high-risk  elements. 

9.  Identify  residual  risks. 

The  next  sections  describe  each  of  these  steps  in  more  detail,  using  the  Business  Considera¬ 
tions:  Competition  strategy  element  as  an  example. 
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5.4.3. 1  Step  1  -  Define  Acquisition  Objectives 

Start  with  a  clear  definition  of  the  objectives  of  the  acquisition.  For  example,  in  the  DoD, 
these  objectives  are  based  on  the  initial  program  inputs  originating  with  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  and  Joint  Requirements  Oversight  Council 
(JROC).  These  objectives  are  also  influenced  by  inputs  from  end  users  and  other  program 
stakeholders.  The  acquisition  objectives  should  be  clearly  documented  and  kept  up-to-date 
throughout  the  acquisition  life  cycle.  These  objectives  form  the  baseline  on  which  the  acqui¬ 
sition  and  the  product  development  will  proceed. 

5.4.3.2  Step  2  -  Identify  and  Evaluate  Drivers 

Now  evaluate  each  of  the  chosen  strategy  drivers  for  your  program.  In  essence,  you  are  mak¬ 
ing  a  judgment  about  the  conditions  of  your  program  in  the  context  of  the  range  of  values 
defined  for  the  strategy  driver.  As  noted  earlier,  this  is  a  subjective  process  intended  to  spur 
discussion  and  to  clarify  thought;  it  is  not  meant  to  provide  a  definitive  and  quantified  as¬ 
sessment  of  program  risk. 

One  effective  method  of  evaluating  drivers  is  to  do  so  as  a  group  within  the  program  office. 
For  example,  relevant  program  office  staff  (the  PM,  deputy  PM,  Chief  Engineer,  Contracting 
Officer,  and  others)  meet  to  discuss  the  program  and  the  strategy  drivers  influencing  it.  Each 
then  completes  an  independent  evaluation  of  all  of  the  strategy  drivers.  Subsequently,  the 
group  reconvenes  and  reconciles  the  independently  generated  evaluations.  Each  strategy 
driver  is  reviewed  and  discussed  to  reach  a  consensus  evaluation.  The  drivers  can  then  be 
marked  on  the  slider  bars,  as  shown  in  Figure  18. 
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Figure  18:  Evaluated  Strategy  Drivers 
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5.4.3. 3  Step  3  -  Decompose  Strategy  into  Elements 

As  noted  in  Section  4. 1,  an  acquisition  strategy  comprises  a  number  of  individual  strategy 
elements.  The  acquisition  planner  must  identify  the  applicable  strategy  elements  based  on  the 
scope  of  the  acquisition.  The  result  is  a  list  of  strategy  elements  similar  to  that  in  Table  5. 

5.4. 3.4  Step  4  -  Rank  Strategic  Choices  by  Risk 

In  Section  5.2.2  we  identified  three  strategy  choices  for  the  Business  Considerations:  Compe¬ 
tition  strategy  element  and  ranked  them  in  terms  of  risk  mitigation  capabilities  as  follows: 

Competition  Strategy  Risk  Mitigation  Capabilities 

Sole  Source  Contracting  Low 

Full  and  Open  Competition  after  Medium 

Exclusion  of  Sources 

Full  and  Open  Competition  High 


The  premise  of  this  ranking  was  that  broader  and  more  open  competition  elicits  a  wider  range 
of  ideas  and  options  for  the  acquirer  to  choose  from,  thereby  providing  greater  opportunities 
for  risk  reduction.  There  are  some  conditions  under  which  this  premise  may  not  hold,  so  be 
sure  to  evaluate  the  rankings  for  your  particular  program. 


5.4. 3. 5  Step  5  -  Identify  Strategy  Drivers 

In  the  notional  example  in  Table  14,  we  find  that  the  Business  Considerations:  Competition 
strategy  element  is  influenced  by  the  following  strategy  drivers: 


Strategy  Driver 

Software  Criticality 

Policies  and  Mandates 

Supplier  Availability 

Mission  Needs  and  Scope 

Funding  Constraints 

Funding  Profile 

Schedule  Constraints 

Urgency 

PMO  Staff  Skills 

PMO  Staff  Capacity 

PMO  Staff  Stability 

Requirements  Volatility 

Requirements  Understanding 

Quality  Attribute  Definition  Quality 

Interoperability 


Level  of  Influence 

Strong  (S) 

Strong  (S) 

Strong  (S) 

Strong  (S) 

Strong  (S) 

Strong  (S) 

Strong  (S) 

Strong  (S) 

Medium  (M) 
Medium  (M) 
Medium  (M) 

Strong  (S) 

Strong  (S) 

Strong  (S) 

Strong  (S) 
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Precedence 

Strong  (S) 

Quality  Attribute  Constraints 

Strong  (S) 

Technology  Readiness 

Strong  (S) 

Legacy  Considerations 

Strong  (S) 

COTS  /  GOTS  /  Reuse 

Strong  (S) 

While  Table  14  identifies  strategy  driver/strategy  element  mappings  for  a  typical  program, 
your  program  may  differ.  It  is  advisable  to  review  all  of  the  strategy  drivers  to  see  if  they  in¬ 
fluence  the  chosen  strategy  element  for  your  program. 

5.4. 3. 6  Step  6  -  Relate  Drivers  to  the  Strategy  Element 

Next,  analyze  each  of  the  drivers  in  terms  of  the  risks  generated  that  can  be  mitigated  by  a 
competitive  strategy  selection.  The  following  paragraphs  represent  such  an  analysis  based  on 
the  Business  Considerations:  Competition  strategy  element  example. 

Software  Criticality  -  Software  is  typically  a  risky  part  of  system  development.  As  such,  the 
more  critical  the  software  is  to  the  program,  the  more  risk  it  imparts  to  the  program.  Elicita¬ 
tion  of  more  ideas  through  wider  competition  provides  some  mitigation  of  this  risk. 

Higher  Software  Criticality  =>  Higher  Project  Risk 

Policies  and  Mandates  -  Policies  and  mandates  applied  to  a  program  restrict  the  latitude  of 
program  decisions  and  often  influence  the  technical  development  of  the  program,  potentially 
forcing  it  into  non-optimal  solutions.  More  competition  may  provide  different  methods  for 
addressing  some  of  the  constraints  imposed  by  policies  and  mandates. 

More  Policies  and  Mandates  =^>  Higher  Project  Risk 

Supplier  Availability  -  A  lack  of  qualified  suppliers  impacts  the  competitive  strategy.  With¬ 
out  several  capable  suppliers,  full  and  open  competition  may  not  be  possible.  Furthermore, 
attempting  full  and  open  competition  in  the  absence  of  qualified  suppliers  simply  invites  bids 
by  unqualified  suppliers;  an  occurrence  that  places  additional  demands  on  the  program  office 
during  source  selection,  but  provides  no  real  benefit  to  the  program.  Alternatively,  the  pres¬ 
ence  of  a  number  of  qualified  suppliers  enables  competition.  Competition  reduces  perform¬ 
ance  risk  by  eliciting  more  ideas  and  more  alternative  solutions.  Competition  also  reduces 
cost  and  schedule  risk  by  encouraging  optimum  performance  from  the  suppliers. 

Fewer  Suppliers  =^>  Higher  Project  Risk 

Mission  Needs  and  Scope  -  Stringent,  non-negotiable  mission  needs  limit  the  flexibility  of 
the  PMO  and  constrain  the  trades  and  technical  alternatives  that  can  be  considered.  This 
drives  the  demands  that  the  PMO  clearly  specify  the  required  performance  in  unambiguous 
and  complete  terms  and  ensure  that  the  supplier  delivers  exactly  that  level  of  performance.  In 
choosing  a  supplier,  the  PMO  must  assess  the  product  performance  risks  of  the  suppliers  pro- 
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posed  approach  to  ensure  that  the  required  performance  will  be  met.  Broader  competition 
helps  the  PMO  accomplish  this  by  providing  a  number  of  alternatives  to  contrast  and  com¬ 
pare  in  the  search  for  the  optimal  solution. 

Likewise,  programs  of  greater  scope  challenge  the  PMO.  Larger  scope  implies  a  greater  chal¬ 
lenge  for  the  supplier,  making  the  PMO’s  choice  of  the  “best”  supplier  more  important. 
Again,  broader  competition  helps  the  PMO  find  the  supplier  best  able  to  handle  the  chal¬ 
lenges  of  a  large  program. 


More  Stringent  Mission  Needs  =^>  Higher  Project  Risk 
Larger  Project  Scope  =^>  Higher  Project  Risk 

Funding  Constraints  -  Rigid  funding  constraints  may  force  the  acquirer  to  choose  less  ca¬ 
pable  suppliers,  to  reduce  program  office  staff  to  marginal  levels,  to  choose  sub-optimal 
technical  solutions,  and  so  on.  Elicitation  of  more  ideas  through  wider  competition  helps  the 
acquirer  to  choose  the  most  capable  supplier  and  the  best  technical  solution  within  the  fund¬ 
ing  constraints. 


Rigid  Funding  Constraints  =^>  Higher  Project  Risk 

Funding  Profile  -  It  is  important  that  the  funding  profile  match  the  needs  of  the  program. 
Inadequate  funding  in  early  phases  of  the  program  may  force  delays  of  critical  aspects  of  the 
development  effort  such  as  requirements  development,  systems  engineering,  trade  studies, 
and  so  on.  Elicitation  of  more  alternatives  through  wider  competition  helps  the  acquirer  to 
choose  the  supplier  with  the  best  approach  for  addressing  funding  limitations.  Additionally, 
competition  in  general  tends  to  drive  prices  down. 

Mismatched  Funding  Profile  =^>  Higher  Project  Risk 

Schedule  Constraints  -  Rigid  schedule  constraints  can  have  negative  impacts  on  all  phases 
of  a  program.  Such  constraints  may  limit  the  acquirer’s  ability  to  produce  a  comprehensive 
and  accurate  solicitation,  forcing  the  acquirer  to  “shortcut”  critical  early  activities  such  as 
requirements  analysis  and  systems  engineering.  The  acquirer  may  also  be  forced  to  reduce 
the  amount  of  time  devoted  to  source  selection,  which  could  result  in  the  choice  of  less  capa¬ 
ble  suppliers  and  sub-optimal  technical  solutions.  Schedule  constraints  imposed  on  the  sup¬ 
plier  may  have  equally  dire  consequences  by  forcing  the  supplier  to  abandon  proven  proc¬ 
esses,  take  shortcuts  in  analysis  and  design,  choose  sub-optimal  solutions,  and  perform 
marginal  or  inadequate  verification. 

The  impact  of  schedule  constraints  on  a  competition  strategy  can  be  viewed  in  two  opposing 
ways.  When  faced  with  a  tight  schedule,  the  acquirer  may  focus  on  reducing  solicitation  and 
award  time  in  order  to  get  a  supplier  on  contract  sooner.  This  creates  a  tendency  toward  using 
sole  source  contracting  to  eliminate  the  time  needed  for  proposal  evaluation  and  source  selec¬ 
tion.  This  may  save  some  time;  however,  it  commits  the  program  to  the  approach  and  the 
schedule  of  just  one  supplier.  Alternatively,  broader  competition  may  require  more  time  to 


CMU/SEI-2006-TR-002 


91 


get  to  contract  award.  However,  if  the  need  for  rapid  delivery  is  stressed  in  the  solicitation 
and  is  a  factor  in  the  contract  award,  the  competing  bidders  will  strive  for  the  shortest  deliv¬ 
ery.  The  time  saved  in  the  execution  phase  of  the  program  often  outweighs  the  time  saved  in 
the  pre-award  phase  of  the  program. 


Rigid  Schedule  Constraints  =^>  Higher  Project  Risk 

Urgency  -  An  urgent  need  for  the  program  deliverable  generates  schedule  pressure.  The 
PMO  must  find  a  supplier  not  only  capable  of  producing  the  required  performance,  but  also 
capable  of  producing  it  quickly  enough.  When  choosing  a  supplier,  the  PMO  must  assess  the 
ability  of  the  bidders  to  meet  the  required  schedule.  Broader  competition  helps  the  PMO  ac¬ 
complish  this  by  providing  a  number  of  alternatives  to  contrast  and  compare  in  the  search  for 
the  supplier  most  capable  of  meeting  the  required  delivery  deadlines. 

High  Urgency  =^>  Higher  Project  Risk 

PMO  Staff  Skills  -  A  less  skillful  PMO  staff  may  face  challenges  managing  stakeholder  in¬ 
volvement,  developing  a  high-quality  solicitation,  evaluating  contractor  proposals,  choosing 
the  best  supplier,  monitoring  and  controlling  supplier  performance,  and  so  on.  All  of  these 
issues  increase  program  risk.  The  interaction  of  this  driver  with  the  formation  of  a  competi¬ 
tive  strategy  is  problematic.  To  assess  this  interaction,  we  will  evaluate  the  impact  of  this 
driver  on  several  competitive  strategies 

•  Full  and  Open  Competition  -  A  less  skilled  PMO  may  faces  challenges  in  the  creation  of 
an  RFP  and  SOW,  in  source  selection,  and  in  contract  negotiations.  Further  challenges  in 
monitoring  and  controlling  the  selected  contractor  and  negotiating  changes  in  cost,  scope, 
and  schedule  also  follow. 

•  Sole  Source  Contracting  -  A  less  skilled  PMO  may  face  challenges  in  the  creation  of  a 
SOW,  in  identifying  the  appropriate  source,  and  in  negotiating  a  contract.  Further  chal¬ 
lenges  in  monitoring  and  controlling  the  selected  contractor  and  negotiating  changes  in 
cost,  scope,  and  schedule  also  follow. 

PMOs  face  similar  challenges  with  these  two  strategies;  both  include  challenges  in  SOW 
creation,  contract  negotiations,  contractor  monitoring  and  control,  and  change  negotiation. 

The  primary  difference  is  in  the  solicitation  and  source  selection  process. 

•  For  full  and  open  competition,  the  PMO  must  generate  an  RFP,  evaluate  proposals,  and 
choose  a  supplier.  This  process  is  defined  in  considerable  detail  in  the  FARs. 

•  For  sole  source  contracting,  the  PMO  must  identify  and  select  a  supplier.  This  process  is 
also  defined  in  considerable  detail  in  the  FARs. 

The  execution  of  a  full  and  open  competition  contract  is  probably  more  challenging  for  a 
PMO  than  the  execution  of  a  sole  source  contract.  On  the  other  hand,  the  success  of  the  pro¬ 
gram  rests  heavily  on  finding  a  capable  and  committed  supplier;  a  result  more  likely  to  be 
attained  with  full  and  open  competition.  Selecting  the  “right”  supplier  probably  poses  a 
greater  risk  than  the  additional  challenges  of  full  and  open  competition  during  solicitation 
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and  source  selection.  Furthermore,  these  solicitation  and  source  selection  challenges  may  be 
mitigated  by  temporarily  augmenting  the  PMO  staff  with  Systems  Engineering  and  Technical 
Assistance  (SETA)  contractors  and  outside  support  services.  The  net  result  of  this  analysis  is 
a  determination  that  a  less  skilled  PMO  staff  encourages  the  use  of  broader  competition. 

Lower  PMO  Staff  Skills  =^>  Higher  Project  Risk 

PMO  Staff  Capacity  -  Inadequate  PMO  staff  capacity  has  much  the  same  impact  as  inade¬ 
quate  PMO  staff  skills.  Through  a  similar  analysis,  we  can  reach  a  similar  conclusion  that 
inadequate  PMO  staff  capacity  encourages  the  use  of  broader  competition. 

Inadequate  PMO  Staff  Capacity  =^>  Higher  Project  Risk 

PMO  Staff  Stability  -  Inadequate  PMO  staff  stability  has  much  the  same  impact  as  inade¬ 
quate  PMO  staff  skills  and  inadequate  staff  capacity.  Through  a  similar  analysis,  we  can 
reach  a  similar  conclusion  that  inadequate  PMO  staff  stability  encourages  the  use  of  broader 
competition. 


Inadequate  PMO  Staff  Capacity  =>  Higher  Project  Risk 

Requirements  Volatility  -  Poorly  defined  and  volatile  requirements  introduce  program  risk. 
Employing  the  broadest  competition  among  the  available  suppliers  elicits  more  ideas  and 
helps  the  acquirer  choose  the  most  capable  supplier  and  the  most  flexible  technical  solution, 
providing  some  mitigation  of  this  risk. 

High  Requirements  Volatility  =^>  Higher  Project  Risk 

Requirements  Understanding  -  Like  high  requirements  volatility,  poorly  understood  re¬ 
quirements  introduce  program  risk.  Employing  the  broadest  competition  among  the  available 
suppliers  elicits  more  ideas  and  helps  the  acquirer  choose  the  most  capable  supplier  and  the 
most  flexible  technical  solution,  providing  some  mitigation  of  this  risk. 

Poor  Requirements  Understanding  =^>  Higher  Project  Risk 

Quality  Attribute  Definition  Quality  -  Poor  quality  attribute  (QA)  definitions  (definitions 
for  non- functional  requirements  such  as  reliability,  supportability,  security,  and  so  on)  intro¬ 
duces  risk  to  both  the  PMO  and  the  supplier.  Unreasonable  QA  definitions  introduce  cost 
risk,  schedule  risk,  and  the  risk  that  these  requirements  may  not  be  achieved.  Absent  QA 
definitions  introduce  the  risk  that  the  product  may  not  meet  the  needs  of  the  users  and  main- 
tamers.  The  challenge  of  developing  a  product  that  meets  poorly  defined  quality  attributes 
requires  a  highly  capable  supplier.  Employing  the  broadest  competition  among  the  available 
suppliers  elicits  more  ideas  and  helps  the  acquirer  choose  the  most  capable  supplier  and  the 
best  technical  solution,  providing  some  mitigation  of  this  risk. 

Poor  Quality  Attribute  Definitions  =>  Higher  Project  Risk 
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Interoperability  -  A  greater  need  for  interoperability  between  your  program  and  other  pro¬ 
grams  and  products  introduces  challenges  for  both  the  PMO  and  the  supplier.  The  supplier 
must  identify  and  manage  all  of  the  inteiprogram  interoperability  needs.  The  supplier  must 
implement  the  interoperability  requirements.  Both  of  these  factors  introduce  risk  into  the 
program.  Employing  the  broadest  competition  among  the  available  suppliers  elicits  more 
ideas  and  helps  the  acquirer  choose  the  most  capable  supplier  and  the  best  technical  solution, 
providing  some  mitigation  of  this  risk. 

Greater  Interoperability  Needs  =^>  Higher  Project  Risk 

Precedence  -  Developing  an  unprecedented  system  (a  system  unlike  any  that  have  been  built 
before)  is  riskier  than  a  precedented  system  (a  system  similar  to  previously  built  ones).  Em¬ 
ploying  the  broadest  competition  among  the  available  suppliers  elicits  more  ideas  and  helps 
the  acquirer  choose  the  most  capable  supplier  and  the  best  technical  solution,  providing  some 
mitigation  of  this  risk. 


Lack  of  System  Precedence  =^>  Higher  Project  Risk 

Quality  Attribute  Constraints  -  Rigid  and  extensive  constraints  on  QAs  such  as  reliability, 
supportability,  security  and  so  on,  introduce  challenge  and  risk  into  a  program.  Employing 
the  broadest  competition  among  the  available  suppliers  elicits  more  ideas  and  helps  the  ac¬ 
quirer  choose  the  most  capable  supplier  and  the  best  technical  solution,  providing  some  miti¬ 
gation  of  this  risk. 


Many  Quality  Attribute  Constraints  =>  Higher  Project  Risk 

Technology  Readiness  -  Reliance  on  immature  technology  may  create  program  risk.  Em¬ 
ploying  the  broadest  competition  among  the  available  suppliers  elicits  more  ideas  and  pro¬ 
vides  some  mitigation  of  this  risk.  Risk  identification  and  mitigation  may  be  stated  source 
selection  criteria  used  to  help  find  the  lowest  risk  approach. 

Lack  of  Technology  Readiness  =^>  Higher  Project  Risk 

Legacy  Considerations  -  Some  systems  must  interoperate  with  legacy  systems  and  some 
must  be  constructed  with  legacy  components  embedded  within  them.  Yet  others  must  provide 
operational  performance  identical  to  a  legacy  system.  When  compared  with  a  “clean-sheet” 
design  effort,  all  of  these  legacy  considerations  introduce  technical  challenges  and  risk  into  a 
program.  Employing  the  broadest  competition  among  the  available  suppliers  elicits  more 
ideas  and  helps  the  acquirer  choose  the  most  capable  supplier  and  the  best  technical  solution 
to  help  provide  some  mitigation  of  this  risk. 

More  Legacy  Considerations  =^>  Higher  Project  Risk 

COTS  /  GOTS  /  Reuse  -  Some  systems  are  designed  using  COTS  or  GOTS  software  com¬ 
ponents.  Some  are  designed  to  incorporate  components  that  have  been  previously  designed 
for  other  systems.  While  reusing  these  components  can  have  significant  benefit  for  the  pro- 
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ject,  it  also  introduces  risks,  in  that  the  system  must  accommodate  the  performance  and  the 
limitations  of  these  components.  Employing  the  broadest  competition  among  the  available 
suppliers  elicits  more  ideas  and  helps  the  acquirer  choose  the  most  capable  supplier  and  the 
best  technical  solution  to  provide  some  mitigation  of  this  risk. 

More  COTS  /  GOTS  /  Reuse  =>  Higher  Project  Risk 

We  now  have  an  indication  of  the  relationship  between  the  strategy  element  and  risk,  and  the 
relationships  between  the  strategy  drivers  and  risk,  summarized  in  Table  16. 


Table  16:  Driver-to-Strategy  Element  Mapping:  Competition 


Low  Risk 

High  Risk 

Strategy  Element  Risk  Mitigation 

Capabilities 

Business  Considerations:  Competition 

Sole  Source 

Full  and  Open 
Competition 

Strategy  Driver  Risk  Generation 

Software  Criticality 

Very  Low 

Very  High 

Policies  and  Mandates 

Low 

High 

Supplier  Availability 

High 

Low 

Mission  Needs  and  Scope 

Flexible 

Rigid 

Funding  Constraints 

Few 

Many 

Funding  Profile 

Matched 

Mismatched 

Schedule  Constraints 

Few 

Many 

Urgency 

Low 

High 

PMO  Staff  Skills 

Strong 

Weak 

PMO  Staff  Capacity 

Adequate 

Inadequate 

PMO  Staff  Stability 

High 

Low 

Requirements  Volatility 

Low 

High 

Requirements  Understanding 

High 

Low 

Quality  Attribute  Definition  Quality 

High 

Low 

Interoperability 

Simple 

Complex 

Precedence 

High 

Low 

Quality  Attribute  Constraints 

Low 

High 

Technology  Readiness 

Mature 

Immature 

Legacy  Considerations 

Low 

High 

COTS  /  GOTS  /  Reuse 

Low 

High 

We  can  express  the  content  of  Table  16  graphically  using  slider  bars,  as  shown  in  Figure  19. 
This  information  enables  us  to  determine  the  assignment  of  endpoints  to  the  driver  strategy 
bars,  with  the  “Low  Risk”  endpoint  assigned  to  the  left  end  of  the  slider  bar  and  the  “High 
Risk”  endpoint  assigned  to  the  right  end  of  the  slider  bar.  Figure  19  also  captures  the  strength 
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of  the  linkage  between  the  driver  and  the  strategy  element;  the  linkages  of  Table  14  (i.e., 
Strong,  Medium,  Weak,  or  None)  are  shown  in  the  column  labeled  “WEIGHT.” 
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ISTRATEGY  ELEMENT:  [Competition 


STRATEGIES 

1 

Full  and  Open  Competition 

2 

3 

4 

Full  and  Open  Competition  After  Exclusion  of  Sources 

5 

6 

7 

Sole  Source  Contracting 

8 

STEP  2: 

Assign  endpoints  for 
each  of  the  relevant 
drivers 


h 


blank  - 1  Ctrl-Shft-~|— 
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Mission  Needs  and  Scope 


Funding  Constraints 


Funding  Profile 


Schedule  Constraints 


PMO  Staff  Skills 


PMO  Staff  Capacity 


PMO  Staff  Stability 


PMO  Process  Focus 


Number  and  Diversity 


Level  of  Engagement 


Level  of  Agreement 


Supplier  Staff  Skills 


Supplier  Staff  Capacity 


Supplier  Staff  Stability 


Supplier  Performance  to 
Date 


Requirements  Volatility 


Requirements 

Understanding 


Quality  Attribute 
Definition  Quality 


Interoperability 


Quality  Attribute 
Constraints 


Technology  Readiness 


Legacy  Considerations 


COTS  /  GOTS  l  Reuse 


Test  Environment 
Complexity 


Test  Environment 
Availability 


Number  of  System 
Configurations 


Number  of  Sites 


RANGE 


Very  Low 


Inadequate 


Inadequate 


Simple 


Very  High 


High 


Rigid 


Many 


Many 


High 


Strong 


Adequate 


Strong 


Large 


High 


High 


Strong 


Adequate 
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High 


High 


Complex 
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Low 
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Low 

Mature 

Low 

Low 


Increasing  Risk 


r 

Strategy  Slider  Bar 

— 4mcr<j..mg  Risk  Mitigation  Capability!-* 

Driver  Slider  Bars 

STEP  1: 

Complete  diivei 
assessments  on 
"Driver 
Assessment" 


STEP  ♦: 

Select  cell  for 
chosen  strategy 
and  press  Ctrl- 


]  STEP  3: 

J  Press  Ctrl-Shft-E 
I  to  transfer  driver 
|  evaluations  from 
"Driver 


High 

Low 

Low 

Complex 

Low 

High 

Immature 

High 

High 


Figure  19:  Slider  Bar  Set:  Competition 
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Note  that  these  relationships  are  subjective.  For  example,  consider  the  “Quality  Attribute 
Constraints”  driver.  We  asserted  that  strong  constraints  on  reliability,  supportability,  and  so 
on,  introduce  challenge  and  complexity  into  the  program,  thereby  creating  a  risk  of  cost  over¬ 
run,  schedule  delays,  and  more.  Thus,  high  “Quality  Attribute  Constraints”  were  equated  with 
high  risk.  An  alternative  and  equally  valid  perspective  could  be  that  strong  constraints  on  re¬ 
liability,  supportability,  and  so  on  increase  the  likelihood  of  satisfying  the  mission  needs  and 
producing  a  fully  capable  product.  In  this  case,  high  “Quality  Attribute  Constraints”  would  be 
equated  with  low  risk. 

5.4.3. 7  Step  7  -  Map  Driver  Evaluations  to  the  Strategy  Element 

Now  that  the  endpoints  have  been  mapped  for  the  selected  strategy  element,  we  can  transfer 
these  evaluations  to  the  driver  slider  bars  for  the  strategy  element,  as  shown  in  Figure  20. 
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ISTRATEGY  ELEMENT:  ICompetition 


STRATEGIES 

1 

Full  and  Open  Competition 

2 

3 

4 

Full  and  Open  Competition  After  Exclusion  of  Sources 
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6 

7 
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Figure  20:  Driver  Evaluations  Mapped  to  Strategy  Element 
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5.4. 3.8  Step  8  -  Choose  Strategy 

As  shown  in  Figure  20,  most  of  the  strategy  drivers  are  marked  between  the  middle  and  the 
right  (high  risk)  side  of  the  slider  bars.  This  indicates  that  the  perceived  amount  of  risk  from 
these  drivers  is  moderate  to  high.  Our  goal  is  to  choose  a  strategy  that  mitigates  the  risks  as 
much  as  possible.  As  an  example,  we  will  choose  the  middle  strategy,  Full  and  Open  Compe¬ 
tition  After  Exclusion  of  Sources.  As  shown  in  Figure  21,  this  strategy  provides  some  risk 
mitigation  for  many,  but  not  all  of  the  drivers.  Drivers  with  evaluations  to  the  right  of  the 
strategy  represent  risks  that  the  strategy  element  is  not  mitigating  to  its  fullest  extent.  These 
risks  need  to  be  addressed  through  other  risk  management  activities.  On  the  other  hand,  many 
of  the  driver  evaluations  are  marked  to  the  left  of  the  chosen  strategy.  We  could  interpret  this 
to  mean  that  the  risks  from  these  drivers  are  “over-mitigated.”  While  this  may  be  beneficial 
from  a  risk  management  perspective,  it  may  not  be  cost-effective;  you  may  be  choosing  a 
more  expensive  strategy  than  is  warranted  for  the  level  of  risk  in  your  program. 
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ISTRATEGY  ELEMENT:  ICompetition 


Figure  21:  Competition  Strategy  #1 
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Alternatively,  if  we  choose  a  Full  and  Open  Competition  strategy,  the  slider  bar  set  appears 
as  shown  in  Figure  22.  The  strategy  selection  means  that  the  maximum  amount  of  risk  miti¬ 
gation  available  from  the  strategy  element  has  been  applied  to  the  risk  arising  from  all  of  the 
drivers.  It  does  not  mean  that  the  risk  is  fully  mitigated.  Now,  all  of  the  drivers  show  exces¬ 
sive  mitigation,  a  situation  that  may  not  be  cost  effective. 
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ISTRATEGY  ELEMENT:  [Competition 


Figure  22:  Competition  Strategy  #2 
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5.4. 3. 9  Step  9  -  Identify  Residual  Risks 

The  strategy  selection  performed  in  Step  5  does  not  necessarily  provide  complete  mitigation 
of  the  risks  arising  from  the  drivers.  Choosing  a  strategy  “to  the  right”  of  the  strategy  driver 
evaluation  does  not  mean  that  the  risk  is  fully  mitigated.  It  merely  means  that  the  maximum 
amount  of  risk  mitigation  available  from  the  strategy  element  has  been  applied  to  that  risk. 
Additional  risk  mitigation  actions  may  be  required  through  the  program’s  risk  mitigation 
process. 

Thus,  after  the  strategy  is  chosen,  the  PMO  must  examine  the  program  to  identify  and  man¬ 
age  the  residual  risks. 

5.4.4  Evaluating  an  Existing  Strategy 

For  a  program  with  an  acquisition  strategy  already  in  execution,  slider  bars  can  be  used  to 
perform  a  strategy  analysis.  Other  than  Step  8,  the  steps  are  similar  to  those  outlined  in  Sec¬ 
tion  5.4.3. 

1.  Define  the  acquisition  objectives. 

2.  Identify  and  evaluate  drivers  the  factors  that  drive  the  program. 

3.  Decompose  the  strategy  into  individual  strategy  elements. 

4.  For  the  selected  strategy  element,  identify  the  potential  strategic  choices,  and  rank  them 
in  order  of  their  risk  mitigation  capabilities  for  your  particular  program. 

5.  Evaluate  the  strategy  drivers  for  the  program  to  identify  those  that  influence  the  strategy 
element  through  the  introduction  of  risk  that  may  be  mitigated  by  strategy  element. 

6.  Define  the  relationship  between  the  risk  generated  by  the  strategy  driver  and  the  risk 
mitigation  capabilities  of  the  strategy  element. 

7.  Map  the  driver  evaluations  from  Step  1  to  the  strategy  element  using  the  slider  bars. 

8.  Indicate  current  strategy. 

9.  Identify  residual  risks. 

Perform  Steps  1-7  exactly  as  described  in  Section  5.4.3.  For  Step  8,  indicate  the  previously 
defined  program  strategy  that  is  currently  in  execution  on  the  strategy  element  slider  bar.  For 
Step  9,  evaluate  the  residual  risks  resulting  from  the  strategy  drivers  “to  the  right”  of  the  cho¬ 
sen  strategy,  as  in  Section  5.4.3.  These  risks  must  then  be  addressed  in  the  program’s  risk 
management  activities. 
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Conclusion 


Developing  an  acquisition  strategy  is  a  process  that  is  not  well  understood.  In  most  cases,  it  is 
performed  in  an  ad  hoc  manner,  without  due  consideration  of  the  internal  and  external  factors 
driving  the  project.  In  crafting  an  acquisition  strategy,  the  program  manager  must  balance 
many  conflicting  objectives,  choosing  a  strategy  that  addresses  some  and  ignores  others. 

The  research  results  presented  in  this  report  support  a  more  systematic  approach  to  reasoning 
about  software  risk  on  a  program.  The  methods  and  techniques  presented  contribute  to  the 
work  focused  on  developing  an  acquisition  strategy  from  a  sound,  systems  engineering  ap¬ 
proach. 

The  report  is  designed  to  help  acquisition  planners  more  systematically 

•  profile  conditions  that  impact  program  risk  (drivers) 

•  identify  sources  of  potential  software  risk  early  in  the  program  and  throughout  the  pro¬ 
gram  life  cycle 

•  develop  an  acquisition  strategy  by  decomposing  it  into  the  strategy  elements  that  com¬ 
pose  it  and  addressing  the  elements  individually  and  then  collectively 

•  reason  about  their  acquisition  strategy’s  ability  to  mitigate  software  risk  in  an  ongoing 
manner 

One  of  the  key  methods  proposed  can  be  used  to  develop  an  acquisition  strategy  by  evaluat¬ 
ing  a  program’s  drivers,  or  it  can  be  used  to  evaluate  an  existing  acquisition  strategy  given 
the  program’s  drivers. 

To  develop  a  strategy  acquisition  planners  would 

1.  Define  the  objectives  of  the  acquisition. 

2.  Identify  and  evaluate  the  factors  that  drive  the  program. 

3.  Decompose  the  strategy  into  individual  strategy  elements. 

4.  For  the  selected  strategy  elements,  identify  the  potential  strategic  choices  and  rank  them 
in  order  of  their  risk  mitigation  capabilities  for  you  program. 

5.  Evaluate  the  strategy  drivers  for  the  program  to  identify  those  that  influence  the  strategy 
element  through  the  introduction  of  risk  that  may  be  mitigated  by  the  strategy  element. 

6.  Define  the  relationship  between  the  risk  generated  by  the  strategy  drive  and  the  risk 
mitigation  capabilities  of  the  strategy  element. 

7.  Map  the  driver  evaluations  from  Step  2  to  the  strategy  element  using  the  slider  bars. 
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8.  Choose  the  strategy  that  best  mitigates  the  risk  elements. 

9.  Identify  residual  risks. 

For  a  program  with  an  acquisition  strategy  already  in  execution,  slider  bars  can  be  used  to 
perform  a  strategy  analysis.  Steps  1-7  are  performed  exactly  as  described  in  Section  5.4.3. 
For  Step  8,  indicate  the  previously  defined  program  strategy  that  is  currently  in  execution  on 
the  strategy  element  slider  bar.  For  Step  9,  evaluate  the  residual  risks  resulting  from  the  strat¬ 
egy  drivers  “to  the  right”  of  the  chosen  strategy,  as  described  in  Section  5.4.3.  These  risks 
must  then  be  addressed  in  the  program’s  risk  management  activities. 

The  ASDT  is  provided  to  help  acquisition  planners  work  through  the  method  and  techniques. 
Acquisition  planners  can  apply  the  method  and  techniques  discussed  in  this  report  without 
using  the  ASDT.  The  ASDT  is  provided  so  that  acquisition  organizations  do  not  have  to  de¬ 
velop  templates  from  scratch.  The  ASDT  is  available  for  download  on  the  SEI  Publications 
Web  site  at  http://www.sei.cmu.edu/publications/publications.html. 

The  research  presented  in  this  report  focused  on  identifying  and  mitigating  software  risk  in  a 
program  during  the  acquisition  planning  process.  Flowever,  the  methods  and  techniques  we 
present  can  be  applied  to  acquisition  planning  areas  other  than  software  risk. 
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Appendix  A  Templates  for  Software  Driver 

Slider  Bars 


I  STRATEGY  ELEMENT:  7 


Figure  23:  Strategy  Slider  Bar  Template 
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I  STRATEGY  ELEMENT:  7 


Figure  24:  Strategy  Slider  Bar  Template  (continued) 
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Appendix  B  Acronyms 


IA 

Information  Assurance 

AP 

Acquisition  Plan 

ASDT 

Acquisition  Strategy  Development  Tool 

BEA 

Business  Enterprise  Architecture 

C4ISR 

Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance  and  Reconnaissance 

CBT 

Computer-Based  Training 

CDD 

Capabilities  Development  Document 

CLS 

Contractor  Fogistics  Support 

CMMI 

Capability  Maturity  Model  Integration 

CMMI-AM 

Capability  Maturity  Model  Integration  -  Acquisition  Module 

COTS 

Commercial  Off- the- Shelf 

CoN 

Certificate  of  Networthiness 

Cto 

Certificate  to  Operate 

DAU 

Defense  Acquisition  University 

DL 

Distance  beaming 

DM 

Data  Management 

DoD 

Department  of  Defense 

ESOH 

Environment,  Safety,  and  Occupational  Health 

FAR 

Federal  Acquisition  Regulation 

FFP 

Firm-Fixed-Price 

F/OS 

Free/Open  Source 
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GFE 

Government  Furnished  Equipment 

GFI 

Government  Furnished  Information 

GOTS 

Government  Off-the-Shelf 

HSI 

Human  Systems  Integration 

IA 

Information  Assurance 

IFB 

Invitation  for  Bid 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

JROC 

Joint  Requirements  Oversight  Council 

JTA 

Joint  Technical  Architecture 

MBT 

Main  Battle  Tank 

MOSA 

Modular  Open  Systems  Approach 

ORD 

Operational  Requirements  Document 

PBL 

Performance  Based  Logistics 

PCA 

Permanent  Change  of  Assignment 

PCS 

Permanent  Change  of  Station 

PM 

Program  Manager 

PMO 

Program  Management  Office 

QA 

Quality  Attribute 

RFI 

Request  for  Information 

RFP 

Request  for  Proposal 

RFQ 

Request  for  Quotation 

SETA 

Systems  Engineering  and  Technical  Assistance 

SLOC 

Source  Lines  of  Code 

SOO 

Statement  of  Objectives 

SOW 

Statement  of  Work 

110 


CMU/SEI-2006-TR-002 


SIPRNet 

TEMP 

UAGV 

XML 


Secret  Internet  Protocol  Router  Network 
Test  and  Evaluation  Master  Plan 
Unmanned  Autonomous  Ground  Vehicle 
Extensible  Markup  Language 
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